Oct 31 15:59:44.098644 ip-10-0-140-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 15:59:44.098654 ip-10-0-140-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 15:59:44.098661 ip-10-0-140-84 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 15:59:44.098913 ip-10-0-140-84 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 15:59:54.244813 ip-10-0-140-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 15:59:54.244845 ip-10-0-140-84 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bda0d8b51ec54ee39b72161a11cad89f --
Oct 31 16:01:52.322718 ip-10-0-140-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 16:01:52.703101 ip-10-0-140-84 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:52.703101 ip-10-0-140-84 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 16:01:52.703101 ip-10-0-140-84 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:52.703101 ip-10-0-140-84 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:01:52.707997 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.707910    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.707997 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.707994    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.707997 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.707998    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.707997 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708002    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.707997 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708005    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708009    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708012    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708015    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708018    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708021    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708024    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708026    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708029    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708032    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708034    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708037    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708040    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708042    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708045    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708047    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708050    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708053    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708056    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.708187 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708059    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708063    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708065    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708068    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708071    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708073    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708076    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708079    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708081    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708084    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708086    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708089    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708092    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708094    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708098    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708101    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708104    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708107    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708110    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708113    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.708656 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708115    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708118    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708121    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708124    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708126    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708129    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708131    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708136    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708139    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708141    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708144    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708146    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708149    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708152    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708155    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708157    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708160    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708163    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708166    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708168    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.709130 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708170    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708173    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708176    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708178    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708181    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708183    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708186    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708189    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708192    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708195    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708198    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708200    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708203    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708206    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708210    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708214    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708218    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708221    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708224    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.709619 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708227    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708231    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708234    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708237    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708240    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708243    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708245    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708247    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708250    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708253    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708255    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708258    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708261    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708265    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708269    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708272    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708275    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708277    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708280    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.710077 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708284    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708287    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708290    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708292    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.708295    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709097    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709104    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709109    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709113    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709117    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709120    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709123    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709126    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709129    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709131    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709135    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709137    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709140    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709143    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.710572 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709146    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709149    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709152    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709154    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709157    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709160    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709162    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709166    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709170    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709173    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709176    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709179    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709182    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709184    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709188    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709190    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709193    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709196    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709198    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.711116 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709201    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709203    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709206    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709208    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709211    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709214    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709216    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709219    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709222    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709224    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709227    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709231    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709234    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709236    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709239    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709241    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709244    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709247    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709249    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.711589 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709252    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709255    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709258    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709261    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709264    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709267    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709269    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709272    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709276    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709278    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709281    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709283    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709286    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709288    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709291    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709293    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709296    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709298    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709301    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709303    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.712046 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709306    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709309    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709311    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709314    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709320    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709324    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709326    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709329    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709332    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709335    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709337    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709340    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709343    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709345    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709348    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709351    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709353    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709356    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709358    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709361    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.712545 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709363    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709366    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709368    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709371    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709373    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709376    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709379    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709381    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709384    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709387    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709390    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709392    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709395    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709399    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709897    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709903    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709907    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709909    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709912    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709915    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.713013 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709918    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709920    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709923    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709925    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709928    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709932    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709935    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709937    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709940    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709943    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709945    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709948    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709951    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709953    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709956    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709959    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709961    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709964    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709966    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709969    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.713499 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709972    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709975    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709977    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709980    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709983    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709985    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709988    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709991    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709994    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709997    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.709999    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710002    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710005    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710007    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710010    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710012    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710015    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710018    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710021    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710023    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.713985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710026    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710029    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710032    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710035    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710037    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710040    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710043    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710046    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710048    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710051    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710054    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710056    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710059    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710061    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710064    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710066    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710069    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710072    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710074    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710077    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.714470 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710079    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710082    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710086    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710089    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710092    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710094    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710098    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710102    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710109    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710111    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710114    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710117    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710120    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710124    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710127    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710129    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710133    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710137    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710141    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.714942 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710143    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710146    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710148    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710151    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710154    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710156    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710159    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710161    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710164    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710166    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710169    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710172    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710174    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710177    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710179    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710182    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710185    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710188    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710191    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.715422 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710194    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.710197    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711074    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711085    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711091    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711096    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711100    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711107    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711112    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711116    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711119    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711122    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711126    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711130    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711133    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711136    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711139    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711142    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711145    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711148    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711152    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711155    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711158    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711162    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 16:01:52.715889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711165    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711169    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711172    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711175    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711178    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711181    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711185    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711188    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711191    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711194    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711199    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711203    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711206    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711209    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711212    2580 flags.go:64] FLAG: --enable-server="true"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711215    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711219    2580 flags.go:64] FLAG: --event-burst="100"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711223    2580 flags.go:64] FLAG: --event-qps="50"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711225    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711228    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711231    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711235    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711238    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711242    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711245    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 31 16:01:52.716523 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711248    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711251    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711254    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711256    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711259    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711262    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711265    2580 flags.go:64] FLAG: --feature-gates=""
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711269    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711272    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711275    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711278    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711281    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711284    2580 flags.go:64] FLAG: --help="false"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711287    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711291    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711295    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711297    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711301    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711304    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711307    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711310    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711313    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711316    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 16:01:52.717145 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711319    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711322    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711325    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711328    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711330    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711333    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711336    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711339    2580 flags.go:64] FLAG: --lock-file=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711342    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711345    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711348    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711354    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711356    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711360    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711363    2580 flags.go:64] FLAG: --logging-format="text"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711365    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711369    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711371    2580 flags.go:64] FLAG: --manifest-url=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711374    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711379    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711382    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711386    2580 flags.go:64] FLAG: --max-pods="110"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711389    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711392    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711395    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 16:01:52.717727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711398    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711401    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711404    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711407    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711414    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711417    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711421    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711424    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711427    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711430    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711433    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711436    2580 flags.go:64] FLAG: --port="10250"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711439    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711442    2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00abf68ac65f53838"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711445    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711448    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711464    2580 flags.go:64] FLAG: --register-node="true"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711470    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711475    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711480    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711483    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711486    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711490    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711493    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711496    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711499    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 16:01:52.718327 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711501    2580 flags.go:64] FLAG: --runonce="false"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711504    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711507    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711510    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711513    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711516    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711520    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711523    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711526    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711529    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711532    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711535    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711538    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711541    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711544    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711547    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711553    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711555    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711558    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711563    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711566    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711568    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711571    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711574    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711577    2580 flags.go:64] FLAG: --v="2"
Oct 31 16:01:52.718977 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711582    2580 flags.go:64] FLAG: --version="false"
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711586    2580 flags.go:64] FLAG: --vmodule=""
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711590    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.711594    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711729    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711733    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711736    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711740    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711743    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711748    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711750    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711753    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711756    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711759    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711763    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711767    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711770    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711774    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711778    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711783    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.719582 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711786    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711789    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711791    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711794    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711797    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711800    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711802    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711805    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711809    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711811    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711814    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711817    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711819    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711822    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711824    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711827    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711830    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711832    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711835    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.720062 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711837    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711840    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711845    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711847    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711850    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711853    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711856    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711859    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711863    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711865    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711868    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711870    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711874    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711877    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711880    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711883    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711885    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711888    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711890    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.720579 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711893    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711896    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711898    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711901    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711904    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711907    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711909    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711912    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711914    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711917    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711919    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711922    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711925    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711927    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711930    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711934    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711936    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711939    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711942    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711944    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.721040 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711947    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711950    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711953    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711956    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711958    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711962    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711965    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711967    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711970    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711972    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711976    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711978    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711981    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711983    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711986    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711989    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711991    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711994    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711996    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.721522 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.711999    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712002    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712005    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712007    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712010    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712013    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712015    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712018    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712022    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712025    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712028    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712031    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712034    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.712061    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712109    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712113    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.721985 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712117    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712120    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712124    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712126    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712129    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712132    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712134    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712137    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712139    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712142    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712144    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712147    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712150    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712152    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712155    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712158    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712160    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712163    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712166    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712168    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.722363 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712171    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712173    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712176    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712178    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712181    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712185    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712188    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712190    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712194    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712196    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712199    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712202    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712205    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712208    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712212    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712215    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712217    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712220    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712223    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.722843 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712225    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712228    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712230    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712233    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712235    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712238    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712240    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712243    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712246    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712248    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712251    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712254    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712256    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712259    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712261    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712264    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712266    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712269    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712273    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712275    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.723284 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712278    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712281    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712283    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712286    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712288    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712291    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712293    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712297    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712300    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712303    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712305    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712308    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712311    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712313    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712317    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712319    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712322    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712324    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712327    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712329    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.723762 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712332    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712334    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712337    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712340    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712342    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712345    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712347    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712350    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712353    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712355    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712360    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712364    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712368    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712372    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712376    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712379    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712382    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712384    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712387    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.724266 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712391    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712394    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712397    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712399    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712402    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.712405    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.719714    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.719730    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.719744    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.719749    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719816    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719821    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719825    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719828    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719831    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719835    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719838    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719841    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719845    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.724777 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719847    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719850    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719853    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719855    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719858    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719861    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719864    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719866    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719869    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719871    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719874    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719877    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719879    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719883    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719891    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719894    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719897    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719900    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719902    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719905    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.725283 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719909    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719912    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719916    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719920    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719924    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719929    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719932    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719935    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719937    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719940    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719943    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719945    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719948    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719950    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719953    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719956    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719958    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719961    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719963    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.725783 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719966    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719968    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719971    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719973    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719976    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719978    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719981    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719983    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719991    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719994    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719997    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.719999    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720002    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720005    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720008    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720011    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720013    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720016    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720018    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720021    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.726233 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720023    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720026    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720028    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720031    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720033    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720036    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720039    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720041    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720044    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720047    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720049    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720052    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720054    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720057    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720059    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720062    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720065    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720068    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720070    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720073    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.726702 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720076    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720084    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720087    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720089    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720092    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720096    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720098    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720101    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720103    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720106    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720109    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720111    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720114    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720116    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720119    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720122    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720124    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.727170 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720127    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.720155    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720354    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720359    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720362    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720365    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720367    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720370    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720373    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720375    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720378    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720381    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720383    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720386    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720388    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720391    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.727578 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720393    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720401    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720405    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720408    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720410    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720413    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720415    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720418    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720421    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720423    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720426    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720429    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720431    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720433    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720436    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720438    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720441    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720444    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720446    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.727965 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720449    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720469    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720472    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720474    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720477    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720480    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720482    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720485    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720488    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720490    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720493    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720495    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720498    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720500    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720503    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720512    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720515    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720518    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720520    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720523    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.728403 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720526    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720528    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720531    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720534    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720536    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720539    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720542    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720544    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720547    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720549    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720552    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720556    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720559    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720562    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720565    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720569    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720573    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720576    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.729098 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720580    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720583    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720586    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720589    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720592    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720595    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720598    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720601    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720603    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720606    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720614    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720617    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720619    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720622    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720624    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720627    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720629    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720631    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720634    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720637    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.729891 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720639    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720642    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720644    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720647    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720650    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720652    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720655    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720657    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720660    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720662    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720665    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720667    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720670    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720672    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720675    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.720698    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:01:52.730628 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720751    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720755    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720759    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720761    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720764    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720766    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720769    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720777    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720780    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720783    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720785    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720788    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720790    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720793    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720796    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720798    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720801    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720803    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720806    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:01:52.731004 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720808    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720811    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720814    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720816    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720819    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720821    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720823    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720826    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720828    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720831    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720834    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720836    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720839    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720841    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720844    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720847    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720849    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720852    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720854    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:01:52.731512 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720857    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720859    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720867    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720870    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720873    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720875    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720878    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720880    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720883    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720885    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720888    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720890    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720893    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720895    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720899    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720901    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720904    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720906    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720909    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720911    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:01:52.731961 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720913    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720916    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720918    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720921    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720923    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720926    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720928    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720931    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720934    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720937    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720942    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720945    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720947    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720950    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720952    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720962    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720966    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720969    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720972    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:01:52.732439 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720975    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720978    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720982    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720984    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720987    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720990    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720992    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720994    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720997    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.720999    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721006    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721009    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721011    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721014    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721016    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721019    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721021    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721024    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721026    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721029    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:01:52.732939 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721033    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721035    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721038    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721042    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721045    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721048    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721050    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721053    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:52.721056    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.721121    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.724813    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.725727    2580 server.go:1014] "Starting client certificate rotation"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.725839    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.725886    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.727657    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:01:52.733411 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.731353    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:01:52.742112 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.742088    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 31 16:01:52.745644 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.745631    2580 log.go:25] "Validated CRI v1 image API"
Oct 31 16:01:52.746521 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.746498    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 16:01:52.748756 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.748736    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e0823901-9792-43b1-a0fc-ec3ba05a5aee:/dev/nvme0n1p4 e4ae8f0b-f78f-4cb3-a43a-f9ba54abd905:/dev/nvme0n1p3]
Oct 31 16:01:52.748803 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.748756    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 16:01:52.753714 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.753598    2580 manager.go:217] Machine: {Timestamp:2026-10-31 16:01:52.752368877 +0000 UTC m=+0.312854672 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101048 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c15c76a26d340b0f17ae030400b44 SystemUUID:ec2c15c7-6a26-d340-b0f1-7ae030400b44 BootID:bda0d8b5-1ec5-4ee3-9b72-161a11cad89f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:df:f2:12:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:df:f2:12:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:f7:66:92:7e:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 16:01:52.753714 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.753712    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 16:01:52.753821 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.753787    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 16:01:52.754814 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.754791    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 16:01:52.754946 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.754816    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-84.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 16:01:52.755034 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.754949    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 16:01:52.755034 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.754957    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 16:01:52.755034 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.754997    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:01:52.755034 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.755015    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:01:52.756110 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.756100    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 16:01:52.760224 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.760213    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 16:01:52.760332 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.760323    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 16:01:52.761306 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761296    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 16:01:52.761337 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761310    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 16:01:52.761337 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761319    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 16:01:52.761337 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761327    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 31 16:01:52.761337 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761335    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 16:01:52.761775 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.761760    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:01:52.763804 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.763788    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 16:01:52.769954 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.769941    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:01:52.770820 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.770808    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 16:01:52.770855 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.770839    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 16:01:52.771021 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771010    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 16:01:52.771050 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771038    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 16:01:52.771080 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771053    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 16:01:52.771107 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771083    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 16:01:52.771107 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771098    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 16:01:52.771155 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771108    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 16:01:52.771558 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771550    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 16:01:52.771589 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771559    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 16:01:52.771589 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771567    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 16:01:52.771589 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771573    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 16:01:52.771589 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771587    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 16:01:52.771713 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.771597    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 16:01:52.772190 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.772180    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 16:01:52.772221 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.772192    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 16:01:52.775840 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.775827    2580 server.go:1263] "Started kubelet"
Oct 31 16:01:52.776018 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.775920    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 16:01:52.776121 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.776101    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 16:01:52.777992 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.777951    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 16:01:52.778131 ip-10-0-140-84 systemd[1]: Started Kubernetes Kubelet.
Oct 31 16:01:52.779820 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.779480    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 16:01:52.783773 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.783757    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 16:01:52.784840 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.784820    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 16:01:52.788409 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.788393    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 16:01:52.788916 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.788903    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 16:01:52.789001 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.788979    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:52.789540 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789520    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 16:01:52.789540 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789525    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 16:01:52.789653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789551    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 16:01:52.789653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789637    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 16:01:52.789653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789643    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 16:01:52.789850 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789832    2580 factory.go:55] Registering systemd factory
Oct 31 16:01:52.789850 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.789849    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 31 16:01:52.790067 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.790050    2580 factory.go:153] Registering CRI-O factory
Oct 31 16:01:52.790124 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.790072    2580 factory.go:223] Registration of the crio container factory successfully
Oct 31 16:01:52.790124 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.790110    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 16:01:52.790124 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.790123    2580 factory.go:103] Registering Raw factory
Oct 31 16:01:52.790264 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.790133    2580 manager.go:1199] Started watching for new ooms in manager
Oct 31 16:01:52.791111 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.791097    2580 manager.go:319] Starting recovery of all containers
Oct 31 16:01:52.802914 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.802897    2580 manager.go:324] Recovery completed
Oct 31 16:01:52.807337 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.807324    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:52.809038 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.809020    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:52.809107 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.809056    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:52.809107 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.809071    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:52.811571 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.811552    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 31 16:01:52.811571 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.811566    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 16:01:52.811702 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.811587    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 16:01:52.811702 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.811614    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:52.812351 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.812330    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:52.812416 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.812362    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:52.812416 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.812378    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:52.813833 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.813813    2580 policy_none.go:50] "Start"
Oct 31 16:01:52.813833 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.813835    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 16:01:52.813968 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.813850    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 16:01:52.815910 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.815893    2580 policy_none.go:44] "Start"
Oct 31 16:01:52.857019 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.856991    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 16:01:52.859330 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859313    2580 manager.go:341] "Starting Device Plugin manager"
Oct 31 16:01:52.859413 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.859343    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 16:01:52.859413 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859353    2580 server.go:87] "Starting device plugin registration server"
Oct 31 16:01:52.859649 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859628    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 16:01:52.859734 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859642    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 16:01:52.859823 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859810    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 16:01:52.859916 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859905    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 16:01:52.859916 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.859916    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 16:01:52.860942 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.860925    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 16:01:52.861034 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.860972    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:52.875692 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.875677    2580 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2612 into the system container \"/system.slice\": <nil>"
Oct 31 16:01:52.908165 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.908139    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-84.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 16:01:52.953706 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.953614    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-84.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 16:01:52.954534 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.953570    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-84.us-west-2.compute.internal.18e3a8a737fb1cd1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-84.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-84.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:52.775797969 +0000 UTC m=+0.336283763,LastTimestamp:2026-10-31 16:01:52.775797969 +0000 UTC m=+0.336283763,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-84.us-west-2.compute.internal,}"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.963250    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.965015    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.965046    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.965059    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.965090    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.971870    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.977566    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.978905    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.978926    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:52.978950    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 16:01:52.980847 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:52.978987    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 16:01:53.039739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.039707    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 16:01:53.039871 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.039794    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4sq2s"
Oct 31 16:01:53.067685 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.067647    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 16:01:53.067831 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.067682    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-84.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 16:01:53.067831 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.067676    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-84.us-west-2.compute.internal.18e3a8a739f654d0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-84.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-84.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-84.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:52.809039056 +0000 UTC m=+0.369524851,LastTimestamp:2026-10-31 16:01:52.809039056 +0000 UTC m=+0.369524851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-84.us-west-2.compute.internal,}"
Oct 31 16:01:53.079349 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.079324    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"]
Oct 31 16:01:53.079463 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.079378    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.080214 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.080198    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.080294 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.080231    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.080294 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.080248    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.082389 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.082377    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.082557 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.082541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.082627 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.082579    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.083092 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083076    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.083177 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083091    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.083177 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083110    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.083177 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083110    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.083177 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083135    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.083177 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.083120    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.085338 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.085325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.085385 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.085353    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.086036 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.086018    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.086108 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.086048    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.086108 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.086061    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.091319 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.091298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.091391 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.091326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.091391 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.091349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.113218 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.113201    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.115233 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.115217    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:53.115769 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.115749    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.115839 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.115821    2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192124 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192124 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192288 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192288 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192193    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192288 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192204    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.192288 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.192193    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.248987 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.248925    2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.316898 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.316876    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:53.317816 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.317796    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:53.317905 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.317826    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:53.317905 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.317841    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:53.317905 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.317871    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.368625 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.368605    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.368700 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.368631    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-84.us-west-2.compute.internal\": node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.418541 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.418517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.419807 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:53.419793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:53.471080 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.471049    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.572097 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.572027    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:53.672467 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:53.672420    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.153422 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.153402    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 16:01:55.153764 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.153562    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 16:01:55.153764 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:55.153543    2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7918a53732334b8eb1150a335f38936-a35c44805d999c48.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-84.us-west-2.compute.internal.18e3a8a739f6b32c\": read tcp 10.0.140.84:53608->3.148.19.237:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-84.us-west-2.compute.internal.18e3a8a739f6b32c  default   4943 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-84.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-84.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-84.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:52 +0000 UTC,LastTimestamp:2026-10-31 16:01:53.080238987 +0000 UTC m=+0.640724786,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-84.us-west-2.compute.internal,}"
Oct 31 16:01:55.200848 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:55.200822    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.301547 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:55.301487    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.400928 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.400900    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:01:55.401572 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:55.401559    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.402032 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.402013    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:01:55.402120 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.402052    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:01:55.402120 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.402075    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:01:55.473424 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.473369    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:54 +0000 UTC" deadline="2028-05-10 23:16:51.671676537 +0000 UTC"
Oct 31 16:01:55.473424 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.473414    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13375h14m56.198265749s"
Oct 31 16:01:55.502036 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:55.502009    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:01:55.528623 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.528602    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:55.534115 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.534100    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:55.617882 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.617816    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:55.641926 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.641906    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:55.673829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.673807    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:01:55.674796 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.674784    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:01:55.725901 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:55.725874    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:01:56.007731 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.007685    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102c36941ddb017d6ed417f277ac7f2a.slice/crio-a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9 WatchSource:0}: Error finding container a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9: Status 404 returned error can't find the container with id a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9
Oct 31 16:01:56.008149 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.008123    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65df5c6b29af4cc29df4b46ea700007.slice/crio-50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2 WatchSource:0}: Error finding container 50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2: Status 404 returned error can't find the container with id 50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2
Oct 31 16:01:56.013348 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.013329    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:01:56.190524 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.190473    2580 apiserver.go:52] "Watching apiserver"
Oct 31 16:01:56.217522 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.217502    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 16:01:56.285979 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.285946    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:01:56.286125 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.286055    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 16:01:56.286388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.286367    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-95f9v","openshift-multus/multus-kwgkk","openshift-network-diagnostics/network-check-target-g5dfz","kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-8hdhf","openshift-multus/network-metrics-daemon-7q6q2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq","openshift-dns/node-resolver-9d2bp","openshift-image-registry/node-ca-jjdf9","openshift-network-operator/iptables-alerter-shkgz","openshift-ovn-kubernetes/ovnkube-node-bmvzx","kube-system/global-pull-secret-syncer-4p5d6","kube-system/konnectivity-agent-n76gt"]
Oct 31 16:01:56.290206 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.290189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.290289 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.290269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:01:56.292236 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.292220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.294306 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.294290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.296861 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.296842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:56.296993 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.296919    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:01:56.298992 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.298975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.301073 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.301055    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.303176 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.303156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.305187 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.305173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.305243 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.305221    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:01:56.308639 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.307609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.310243 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.310224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.312550 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.312531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.314905 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.314886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.318350 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.318333    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 16:01:56.329126 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.329208 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.329208 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4xk\" (UniqueName: \"kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.329277 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.329343 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329343 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329343 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.329429 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.329429 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.329486 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.329486 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.329486 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.329569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.329569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329672 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329672 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.329672 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btk9z\" (UniqueName: \"kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.329763 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.329763 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.329763 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.329848 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.329848 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.329904 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.329904 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.329981 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.329981 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.329981 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330126 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.329971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330126 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330126 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330238 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330238 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330238 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s466z\" (UniqueName: \"kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.330388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6mn\" (UniqueName: \"kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.330388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwqj\" (UniqueName: \"kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.330388 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.330932 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.330997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shb5f\" (UniqueName: \"kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk7d8\" (UniqueName: \"kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.331328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.331786 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.331735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.340529 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.340511    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.340529 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.340525    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.340665 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.340511    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 16:01:56.340665 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.340562    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\""
Oct 31 16:01:56.342316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.342300    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 16:01:56.342408 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.342391    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.342463 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.342445    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 16:01:56.342463 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.342450    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\""
Oct 31 16:01:56.342561 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.342492    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.346801 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.346784    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 16:01:56.346888 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.346881    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 16:01:56.347345 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.347330    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\""
Oct 31 16:01:56.348755 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.348743    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.349012 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.349000    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.349059 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.349050    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\""
Oct 31 16:01:56.351039 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.351024    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.351039 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.351036    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.351128 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.351041    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\""
Oct 31 16:01:56.358779 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.358763    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.358864 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.358829    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 16:01:56.358864 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.358853    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.358943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.358863    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 16:01:56.358943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.358853    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\""
Oct 31 16:01:56.380720 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.380703    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.380988 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.380975    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 16:01:56.381262 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.381248    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 16:01:56.381316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.381261    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.381363 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.381349    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 16:01:56.381363 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.381360    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 16:01:56.381428 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.381354    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 16:01:56.384728 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.384712    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\""
Oct 31 16:01:56.386980 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.386965    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\""
Oct 31 16:01:56.387047 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.386989    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 16:01:56.387047 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.387026    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 16:01:56.387047 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.387041    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\""
Oct 31 16:01:56.387047 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.387047    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzfbq"
Oct 31 16:01:56.387264 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.387095    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 16:01:56.387503 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.387490    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 16:01:56.413240 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.411943    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerStarted","Data":"50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2"}
Oct 31 16:01:56.413575 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.413549    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"102c36941ddb017d6ed417f277ac7f2a","Type":"ContainerStarted","Data":"a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9"}
Oct 31 16:01:56.432681 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432757    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432773    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.432784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432779    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432905    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432942    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432940    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.432972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433018    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433067    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.433065 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk9z\" (UniqueName: \"kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433101    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.433121    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433113    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433122    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433170    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.433179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:56.933160432 +0000 UTC m=+3.065482801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433173    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433256    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433296    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.433829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433374    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433375    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433419    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433449    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433523    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433558    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s466z\" (UniqueName: \"kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433568    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.434686 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433637    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6mn\" (UniqueName: \"kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433665    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwqj\" (UniqueName: \"kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433818    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433924    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.435491 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.433993    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434008    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434095    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434165    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434224    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shb5f\" (UniqueName: \"kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7d8\" (UniqueName: \"kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.436342 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434482    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434501    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434548    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434559    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434594    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434657    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.436943 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4xk\" (UniqueName: \"kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435092    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435135    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435187    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435233    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435268    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.434671    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.437569 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435319    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435586    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435739    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435776    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435809    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435834    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435859    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435910    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435939    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.435966    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.436011    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.436080    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:01:56.936066518 +0000 UTC m=+3.068388874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.436101    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.436152    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437132    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437137    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437198    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.438209 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437199    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437224    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437224    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437283    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437288    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437402    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437828    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.437885    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.438082    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.438739 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.438252    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.486120 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.486071    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:56.486120 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.486094    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:56.486120 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.486106    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:56.486340 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.486161    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:56.986146088 +0000 UTC m=+3.118468461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:56.486530 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.486481    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk9z\" (UniqueName: \"kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.487022 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.487005    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.487271 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.487250    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwqj\" (UniqueName: \"kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.487352 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.487331    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6mn\" (UniqueName: \"kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.487656 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.487641    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s466z\" (UniqueName: \"kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.488118 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.488098    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7d8\" (UniqueName: \"kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.488428 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.488411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.488693 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.488675    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shb5f\" (UniqueName: \"kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.490144 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.490128    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4xk\" (UniqueName: \"kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.604845 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.604825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:01:56.609888 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.609875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kwgkk"
Oct 31 16:01:56.614720 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.614700    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5bd675_1035_4a49_af88_1f600d0bd7b7.slice/crio-e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202 WatchSource:0}: Error finding container e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202: Status 404 returned error can't find the container with id e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202
Oct 31 16:01:56.618254 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.618236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:01:56.619191 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.619173    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f20fe20_7b1b_4a7f_a855_6520f94f9603.slice/crio-cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688 WatchSource:0}: Error finding container cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688: Status 404 returned error can't find the container with id cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688
Oct 31 16:01:56.621804 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.621786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:01:56.627045 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.627025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:01:56.627341 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.627323    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a5a550_ba91_4cd4_92a7_bb61db0e3eb5.slice/crio-ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382 WatchSource:0}: Error finding container ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382: Status 404 returned error can't find the container with id ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382
Oct 31 16:01:56.629701 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.629678    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7fd9a4_5924_4e14_b744_d9b11ea3f696.slice/crio-e2054a2a189afaf349f61d56b437a200951ec447c797cd39c3a22a1d098c5a0a WatchSource:0}: Error finding container e2054a2a189afaf349f61d56b437a200951ec447c797cd39c3a22a1d098c5a0a: Status 404 returned error can't find the container with id e2054a2a189afaf349f61d56b437a200951ec447c797cd39c3a22a1d098c5a0a
Oct 31 16:01:56.635326 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.635300    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ec4a55_0dd7_41f6_8837_eac4e3d5cb57.slice/crio-0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07 WatchSource:0}: Error finding container 0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07: Status 404 returned error can't find the container with id 0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07
Oct 31 16:01:56.643663 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.643649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:01:56.649283 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.649268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:01:56.651638 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.651604    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bfbeb1_ecc8_479f_a02e_c6b1d55daaea.slice/crio-2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c WatchSource:0}: Error finding container 2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c: Status 404 returned error can't find the container with id 2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c
Oct 31 16:01:56.656648 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.656628    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4f1c13_63af_4f5b_b0b5_57dab70784af.slice/crio-9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd WatchSource:0}: Error finding container 9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd: Status 404 returned error can't find the container with id 9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd
Oct 31 16:01:56.657477 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.657462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:01:56.665683 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.665661    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode201d4ab_e257_44f9_8595_06552d621cdd.slice/crio-98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c WatchSource:0}: Error finding container 98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c: Status 404 returned error can't find the container with id 98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c
Oct 31 16:01:56.669727 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.669711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:01:56.677996 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:01:56.677979    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb58743c_05b8_4d94_b8d0_84a80eb9c06b.slice/crio-12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e WatchSource:0}: Error finding container 12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e: Status 404 returned error can't find the container with id 12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e
Oct 31 16:01:56.938159 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.938093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:56.938159 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:56.938132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:56.938309 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.938238    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:56.938309 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.938288    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:01:57.938275918 +0000 UTC m=+4.070598272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:56.938309 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.938238    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:56.938402 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:56.938324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:57.938315233 +0000 UTC m=+4.070637597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.038753 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.038720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:57.038888 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.038861    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:57.038888 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.038882    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:57.038972 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.038894    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:57.038972 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.038954    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:58.038937644 +0000 UTC m=+4.171260040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:57.390733 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.390625    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:56 +0000 UTC" deadline="2028-04-19 22:35:10.936303186 +0000 UTC"
Oct 31 16:01:57.390733 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.390679    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12870h33m13.545627153s"
Oct 31 16:01:57.407989 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.407956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:57.408154 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.408121    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:01:57.416547 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.416487    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwgkk" event={"ID":"8f20fe20-7b1b-4a7f-a855-6520f94f9603","Type":"ContainerStarted","Data":"cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688"}
Oct 31 16:01:57.418322 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.418296    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shkgz" event={"ID":"5c5bd675-1035-4a49-af88-1f600d0bd7b7","Type":"ContainerStarted","Data":"e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202"}
Oct 31 16:01:57.419600 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.419573    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq" event={"ID":"e201d4ab-e257-44f9-8595-06552d621cdd","Type":"ContainerStarted","Data":"98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c"}
Oct 31 16:01:57.420682 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.420656    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjdf9" event={"ID":"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea","Type":"ContainerStarted","Data":"2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c"}
Oct 31 16:01:57.421857 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.421829    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9d2bp" event={"ID":"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57","Type":"ContainerStarted","Data":"0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07"}
Oct 31 16:01:57.422911 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.422888    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n76gt" event={"ID":"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5","Type":"ContainerStarted","Data":"ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382"}
Oct 31 16:01:57.424219 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.424195    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e"}
Oct 31 16:01:57.425605 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.425582    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerStarted","Data":"9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd"}
Oct 31 16:01:57.427263 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.427223    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hdhf" event={"ID":"4c7fd9a4-5924-4e14-b744-d9b11ea3f696","Type":"ContainerStarted","Data":"e2054a2a189afaf349f61d56b437a200951ec447c797cd39c3a22a1d098c5a0a"}
Oct 31 16:01:57.944445 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.944210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:57.944445 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:57.944293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:57.944445 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.944421    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.944754 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.944475    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:01:59.944459542 +0000 UTC m=+6.076781908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:57.944754 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.944694    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:57.944754 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:57.944762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:59.944733401 +0000 UTC m=+6.077055765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:58.044933 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:58.044895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:58.045156 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.045117    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:01:58.045156 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.045143    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:01:58.045239 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.045158    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:58.045239 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.045218    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:00.045201181 +0000 UTC m=+6.177523546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:01:58.391380 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:58.391295    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:56 +0000 UTC" deadline="2028-07-09 19:30:27.838071782 +0000 UTC"
Oct 31 16:01:58.391380 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:58.391333    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14811h28m29.446742532s"
Oct 31 16:01:58.412690 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:58.411955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:58.412690 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.412077    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:01:58.412690 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:58.412452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:58.412690 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:58.412552    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:01:59.407544 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:59.407516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:01:59.408203 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:59.407645    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:59.964901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:01:59.965179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:59.965185    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:59.965259    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:03.965241001 +0000 UTC m=+10.097563362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:59.965286    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:01:59.965368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:01:59.965363    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:03.965346597 +0000 UTC m=+10.097668964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:00.066828 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.066794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:00.067482 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.066988    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:00.067482 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.067018    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:00.067482 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.067032    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:00.067482 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.067093    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:04.067076032 +0000 UTC m=+10.199398393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:00.410899 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.410880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:00.411341 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.411007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:00.411341 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.411054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:00.411341 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:00.411103    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:00.454440 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.454393    2580 generic.go:356] "Generic (PLEG): container finished" podID="b65df5c6b29af4cc29df4b46ea700007" containerID="e9b57f3ec02ee4891218672c519f710b96fc898b3c49d1eabfc352bbd373cf21" exitCode=0
Oct 31 16:02:00.454541 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.454478    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerDied","Data":"e9b57f3ec02ee4891218672c519f710b96fc898b3c49d1eabfc352bbd373cf21"}
Oct 31 16:02:00.461754 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:00.461732    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"102c36941ddb017d6ed417f277ac7f2a","Type":"ContainerStarted","Data":"b6eb9ea0adaa255b2c741e8db7054a42d7c9c03dc189ff4bac9937037728065a"}
Oct 31 16:02:01.407923 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.407764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:01.407923 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:01.407889    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:01.473725 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.473252    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n76gt" event={"ID":"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5","Type":"ContainerStarted","Data":"cf126580d30c36dd9fbc42ec5aab5946ac395c528ba90cb36c9f9b14e1bfa053"}
Oct 31 16:02:01.480481 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.480334    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="98926a40b292700ef0aac37f7a67eb12e714229ac64f8ef372913aaebf2954b5" exitCode=0
Oct 31 16:02:01.480481 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.480431    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"98926a40b292700ef0aac37f7a67eb12e714229ac64f8ef372913aaebf2954b5"}
Oct 31 16:02:01.495351 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.495264    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerStarted","Data":"25b3f6d03726bfcdfc5fad2534cd3cc1d623e460808381b204e59e426a3d9460"}
Oct 31 16:02:01.578113 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.578054    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal" podStartSLOduration=6.578034858 podStartE2EDuration="6.578034858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:00.665128009 +0000 UTC m=+6.797450382" watchObservedRunningTime="2026-10-31 16:02:01.578034858 +0000 UTC m=+7.710357232"
Oct 31 16:02:01.683860 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.683765    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n76gt" podStartSLOduration=3.324630291 podStartE2EDuration="7.683748043s" totalImagesPullingTime="4.359117752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:01.579441502 +0000 UTC m=+7.711763879" watchObservedRunningTime="2026-10-31 16:02:01.683748043 +0000 UTC m=+7.816070415"
Oct 31 16:02:01.797599 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:01.797539    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" podStartSLOduration=6.79751708 podStartE2EDuration="6.79751708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:01.684123221 +0000 UTC m=+7.816445592" watchObservedRunningTime="2026-10-31 16:02:01.79751708 +0000 UTC m=+7.929839455"
Oct 31 16:02:02.409979 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:02.407535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:02.409979 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:02.407576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:02.409979 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:02.407665    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:02.409979 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:02.407795    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:03.408133 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:03.408052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:03.408665 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:03.408184    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:03.502192 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:03.502156    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jjdf9" event={"ID":"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea","Type":"ContainerStarted","Data":"6f882d90c684e6e0333017559e78a2441a2379c21310960b7da52bf6fd33f35f"}
Oct 31 16:02:03.506164 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:03.506138    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq" event={"ID":"e201d4ab-e257-44f9-8595-06552d621cdd","Type":"ContainerStarted","Data":"324a9db6673a255dfbdb6fa3d5beea378cb8bc6f2f25ba9deea48c74cf03c0e5"}
Oct 31 16:02:03.607537 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:03.607467    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jjdf9" podStartSLOduration=3.814284241 podStartE2EDuration="9.607451952s" totalImagesPullingTime="5.793167711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:03.606767419 +0000 UTC m=+9.739089792" watchObservedRunningTime="2026-10-31 16:02:03.607451952 +0000 UTC m=+9.739774324"
Oct 31 16:02:04.007449 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.007413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:04.007635 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.007475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:04.007717 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.007696    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:04.007773 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.007767    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.007747165 +0000 UTC m=+18.140069515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:04.008198 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.008174    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:04.008303 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.008228    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.008213098 +0000 UTC m=+18.140535449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:04.108300 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.108248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:04.108475 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.108432    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:04.108475 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.108455    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:04.108475 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.108468    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:04.108650 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.108530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:12.108513148 +0000 UTC m=+18.240835512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:04.408405 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.408332    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:04.408896 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.408510    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:04.409662 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.409636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:04.409770 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:04.409736    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:04.634315 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:04.633735    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:05.078938 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.078908    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:05.079572 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.079380    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:05.102328 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.102309    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 16:02:05.301851 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.301744    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:05.102326179Z","UUID":"692c7555-c072-4db8-9bf2-ba373c6c7a02","Handler":null,"Name":"","Endpoint":""}
Oct 31 16:02:05.303368 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.303349    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 16:02:05.303476 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.303375    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 16:02:05.407760 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.407739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:05.407866 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:05.407844    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:05.516993 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.516915    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9d2bp" event={"ID":"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57","Type":"ContainerStarted","Data":"1563d5e36d012d0c108ee5927e7f6fee6ea9bd588f8bdea52f33739653101396"}
Oct 31 16:02:05.520141 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.519681    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kwgkk" event={"ID":"8f20fe20-7b1b-4a7f-a855-6520f94f9603","Type":"ContainerStarted","Data":"13b442e728d7d441670b18b4e3f6f0842754c7b2db886ff9b38eb2d6f8baef1b"}
Oct 31 16:02:05.522660 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.522611    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq" event={"ID":"e201d4ab-e257-44f9-8595-06552d621cdd","Type":"ContainerStarted","Data":"4d5a163892657de7dd5764a9aaf86380be30203e4c24eea642bddb6e360d15a4"}
Oct 31 16:02:05.523408 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.523387    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:05.727007 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.726953    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9d2bp" podStartSLOduration=3.228363766 podStartE2EDuration="11.726939915s" totalImagesPullingTime="8.498576149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:05.62274677 +0000 UTC m=+11.755069143" watchObservedRunningTime="2026-10-31 16:02:05.726939915 +0000 UTC m=+11.859262286"
Oct 31 16:02:05.833293 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:05.833194    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kwgkk" podStartSLOduration=3.763536793 podStartE2EDuration="11.833180244s" totalImagesPullingTime="8.069643451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:05.83276333 +0000 UTC m=+11.965085706" watchObservedRunningTime="2026-10-31 16:02:05.833180244 +0000 UTC m=+11.965502616"
Oct 31 16:02:06.410075 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:06.409790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:06.410226 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:06.410146    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:06.410338 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:06.409956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:06.410460 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:06.410435    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:06.528458 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:06.526740    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="cc285be1a9deee4b33823082c44648d697047b39442ebbbfc01bfc42caa82c71" exitCode=0
Oct 31 16:02:06.528458 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:06.527575    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"cc285be1a9deee4b33823082c44648d697047b39442ebbbfc01bfc42caa82c71"}
Oct 31 16:02:07.408193 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:07.408162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:07.408346 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:07.408274    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:07.530393 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:07.530359    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shkgz" event={"ID":"5c5bd675-1035-4a49-af88-1f600d0bd7b7","Type":"ContainerStarted","Data":"1f2250967c4b0671225787d98873bbf81ea872a5c3cf0b50f90875eccbce834e"}
Oct 31 16:02:07.533042 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:07.533018    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq" event={"ID":"e201d4ab-e257-44f9-8595-06552d621cdd","Type":"ContainerStarted","Data":"16d6956d5d12c58a59e9d3e28c1513d813b767537db3ae9a7240ff32ce076d8b"}
Oct 31 16:02:07.635103 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:07.635041    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-shkgz" podStartSLOduration=4.861313979 podStartE2EDuration="13.635024886s" totalImagesPullingTime="8.773710907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:07.6342565 +0000 UTC m=+13.766578892" watchObservedRunningTime="2026-10-31 16:02:07.635024886 +0000 UTC m=+13.767347258"
Oct 31 16:02:07.742989 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:07.742882    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq" podStartSLOduration=4.151180966 podStartE2EDuration="13.742862941s" totalImagesPullingTime="9.591681975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:07.742141261 +0000 UTC m=+13.874463633" watchObservedRunningTime="2026-10-31 16:02:07.742862941 +0000 UTC m=+13.875185314"
Oct 31 16:02:08.407847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.407751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:08.407847 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.407759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:08.408078 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:08.407900    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:08.408078 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:08.407930    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:08.539313 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.539278    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="aae147eadb04cf107178d52bb17b2ad13474a3d880a1798fe1bb06f585f12114" exitCode=0
Oct 31 16:02:08.539889 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.539366    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"aae147eadb04cf107178d52bb17b2ad13474a3d880a1798fe1bb06f585f12114"}
Oct 31 16:02:08.541289 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.541206    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hdhf" event={"ID":"4c7fd9a4-5924-4e14-b744-d9b11ea3f696","Type":"ContainerStarted","Data":"d703ec5fb65a4b4270869ae106226eb025437c29224cc2c38a49a6644d78e665"}
Oct 31 16:02:08.799956 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:08.799855    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8hdhf" podStartSLOduration=3.971563078 podStartE2EDuration="14.79984023s" totalImagesPullingTime="10.828277152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:08.799458441 +0000 UTC m=+14.931780814" watchObservedRunningTime="2026-10-31 16:02:08.79984023 +0000 UTC m=+14.932162602"
Oct 31 16:02:09.408282 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:09.408256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:09.408449 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:09.408367    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:10.407693 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:10.407485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:10.408241 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:10.407493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:10.408241 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:10.407841    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:10.408241 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:10.407931    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:10.551991 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:10.551866    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="b21e50682622680997a1d3800c42ae9537f30fb034992ba199d1de825cdbcd01" exitCode=0
Oct 31 16:02:10.551991 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:10.551933    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"b21e50682622680997a1d3800c42ae9537f30fb034992ba199d1de825cdbcd01"}
Oct 31 16:02:11.408105 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:11.408072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:11.408473 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:11.408162    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.064802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.064856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.064975    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.065034    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.065016034 +0000 UTC m=+34.197338404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.065105    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:12.065171 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.065143    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret podName:a029ca40-3196-4f27-b72a-1d2a2b76197e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.065130434 +0000 UTC m=+34.197452799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret") pod "global-pull-secret-syncer-4p5d6" (UID: "a029ca40-3196-4f27-b72a-1d2a2b76197e") : object "kube-system"/"original-pull-secret" not registered
Oct 31 16:02:12.165261 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.165106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:12.165368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.165227    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 16:02:12.165368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.165342    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 16:02:12.165368 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.165357    2580 projected.go:196] Error preparing data for projected volume kube-api-access-5bf7v for pod openshift-network-diagnostics/network-check-target-g5dfz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:12.165578 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.165409    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v podName:55552918-3517-4b42-8e9b-b69bbe124e03 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.165391686 +0000 UTC m=+34.297714036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bf7v" (UniqueName: "kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v") pod "network-check-target-g5dfz" (UID: "55552918-3517-4b42-8e9b-b69bbe124e03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 16:02:12.408646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.407927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:12.408646 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.408090    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:12.408646 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.408136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:12.408646 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:12.408212    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:12.560201 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560126    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"3d60a679c7bca7a15e6110bad2c7d39c7551a1c79f02fc13e1e559e0d257caf7"}
Oct 31 16:02:12.560316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560203    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"4650c5b00597f024069f368ba5f8309742f0a22ad9e38b72b91f6b60e54351a6"}
Oct 31 16:02:12.560316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560234    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"193dd026cd662329fa288b936d5ac81d49c16e5026e03239a2f93a5372a01572"}
Oct 31 16:02:12.560316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560264    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"b7a26c8abf68eaa8ae3b4e4310dbdd8f6f2163afe3955212dd48911fe1f43b66"}
Oct 31 16:02:12.560316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560290    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"91d90e5e9dd1aed2ef15002e960fb43319b3ffbe03dfc6833cefe18e855ab2c0"}
Oct 31 16:02:12.560464 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:12.560322    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"241fba619699650570839c0529d51d87a1462ad197ec19efc9fa28a9c026bcc1"}
Oct 31 16:02:13.407303 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:13.407273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:13.407449 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:13.407364    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:13.564502 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:13.564471    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="b68de7e1e5abe05032ca3d19fd20585d88601089cbfd1a861e54d1e9314aee85" exitCode=0
Oct 31 16:02:13.564940 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:13.564516    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"b68de7e1e5abe05032ca3d19fd20585d88601089cbfd1a861e54d1e9314aee85"}
Oct 31 16:02:14.408055 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:14.408032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:14.408182 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:14.408158    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:14.408254 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:14.408193    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:14.408254 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:14.408245    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:14.569784 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:14.569752    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"079e35191800a811e1a18687007890b350d7174066244779f1372e7fbef3f18f"}
Oct 31 16:02:14.572500 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:14.572476    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="a9ade5cc51d64bd151389a2b10b6381e2acd3dd4e88a34e52e698c8e0f6f267e" exitCode=0
Oct 31 16:02:14.572590 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:14.572521    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerDied","Data":"a9ade5cc51d64bd151389a2b10b6381e2acd3dd4e88a34e52e698c8e0f6f267e"}
Oct 31 16:02:15.407881 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:15.407850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:15.408046 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:15.407945    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:15.578973 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:15.578942    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95f9v" event={"ID":"cd4f1c13-63af-4f5b-b0b5-57dab70784af","Type":"ContainerStarted","Data":"4c79360a557e8bd06a94d50ff53eb987e77aad0e5cb200f6b0623046adf626a7"}
Oct 31 16:02:15.688303 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:15.688226    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-95f9v" podStartSLOduration=8.631607594 podStartE2EDuration="21.688214532s" totalImagesPullingTime="13.056606938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:15.687677191 +0000 UTC m=+21.819999563" watchObservedRunningTime="2026-10-31 16:02:15.688214532 +0000 UTC m=+21.820536902"
Oct 31 16:02:16.407429 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:16.407398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:16.407429 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:16.407421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:16.407698 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:16.407513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:16.407698 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:16.407653    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:16.585076 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:16.585047    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" event={"ID":"eb58743c-05b8-4d94-b8d0-84a80eb9c06b","Type":"ContainerStarted","Data":"9203697e6ce3a9fe6428da9c66a74d2833710bce9cd6a6a080deb452c0380209"}
Oct 31 16:02:16.836228 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:16.836119    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx" podStartSLOduration=7.576862673 podStartE2EDuration="22.836105723s" totalImagesPullingTime="15.25924305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:16.789544399 +0000 UTC m=+22.921866791" watchObservedRunningTime="2026-10-31 16:02:16.836105723 +0000 UTC m=+22.968428152"
Oct 31 16:02:17.407551 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.407527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:17.407726 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:17.407642    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:17.586849 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.586822    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:17.587299 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.586890    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:17.587299 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.586920    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:17.601776 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.601752    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:17.602703 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:17.602268    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:18.408361 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:18.408331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:18.408580 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:18.408477    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:18.408580 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:18.408525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:18.408661 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:18.408602    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:19.078885 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.078817    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4p5d6"]
Oct 31 16:02:19.079279 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.078912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:19.079279 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:19.078991    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:19.082300 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.082270    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5dfz"]
Oct 31 16:02:19.082434 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.082403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:19.082527 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:19.082506    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:19.082914 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.082895    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7q6q2"]
Oct 31 16:02:19.082992 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:19.082972    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:19.083078 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:19.083057    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:20.410466 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:20.410439    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:20.410914 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:20.410559    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:21.408274 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:21.408195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:21.408274 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:21.408228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:21.408435 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:21.408306    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:21.408435 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:21.408377    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:22.408387 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:22.408358    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:22.408889 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:22.408493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:23.408158 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:23.408123    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:23.408158 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:23.408157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:23.408342 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:23.408240    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5dfz" podUID="55552918-3517-4b42-8e9b-b69bbe124e03"
Oct 31 16:02:23.408380 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:23.408341    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4p5d6" podUID="a029ca40-3196-4f27-b72a-1d2a2b76197e"
Oct 31 16:02:24.408413 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.408244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:24.408883 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:24.408467    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7q6q2" podUID="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4"
Oct 31 16:02:24.666022 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.665952    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeReady"
Oct 31 16:02:24.666144 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.666045    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 16:02:24.771128 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.771095    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xwx8c"]
Oct 31 16:02:24.776166 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.776140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.816189 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.816165    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bvb4g"]
Oct 31 16:02:24.819754 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.819738    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xwx8c"]
Oct 31 16:02:24.819835 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.819824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:24.824759 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.824739    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 16:02:24.824877 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.824840    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 16:02:24.824945 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.824905    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\""
Oct 31 16:02:24.859017 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.858991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.859017 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.859019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.859141 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.859040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.859141 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.859126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vcj\" (UniqueName: \"kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.862335 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.862316    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bvb4g"]
Oct 31 16:02:24.925246 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.925182    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 16:02:24.925360 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.925265    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 16:02:24.925360 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.925271    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 16:02:24.925360 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.925335    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\""
Oct 31 16:02:24.959626 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.959728 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.959728 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggz98\" (UniqueName: \"kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:24.959728 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.959839 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:24.959762    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:24.959839 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:24.959839 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vcj\" (UniqueName: \"kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.959839 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:24.959833    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.459813134 +0000 UTC m=+31.592135511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:24.960001 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.959943    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:24.960250 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:24.960234    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:25.014505 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.014477    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vcj\" (UniqueName: \"kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:25.061002 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.060976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggz98\" (UniqueName: \"kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:25.061116 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.061027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:25.063226 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.063208    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:25.113820 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.113796    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggz98\" (UniqueName: \"kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:25.131603 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.131581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:25.408057 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.408031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:25.408057 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.408051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:25.456880 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.456860    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 16:02:25.456880 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.456883    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 16:02:25.457256 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.457068    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\""
Oct 31 16:02:25.457256 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.457083    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 16:02:25.457256 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.457134    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\""
Oct 31 16:02:25.463420 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.463403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:25.463507 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:25.463496    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:25.463549 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:25.463541    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:26.463529489 +0000 UTC m=+32.595851838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:25.548829 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.548762    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bvb4g"]
Oct 31 16:02:25.552822 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:02:25.552801    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e4b8e3_1495_49dd_932a_0bbc5c6b7183.slice/crio-009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd WatchSource:0}: Error finding container 009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd: Status 404 returned error can't find the container with id 009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd
Oct 31 16:02:25.602725 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:25.602704    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bvb4g" event={"ID":"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183","Type":"ContainerStarted","Data":"009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd"}
Oct 31 16:02:26.407596 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:26.407565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:26.456131 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:26.456103    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 16:02:26.456260 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:26.456181    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\""
Oct 31 16:02:26.469376 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:26.469357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:26.469682 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:26.469454    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:26.469682 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:26.469496    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:28.469484303 +0000 UTC m=+34.601806653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:28.080444 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.080413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:28.080841 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.080473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:28.080841 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:28.080569    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:28.080841 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:28.080659    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:03:00.08063787 +0000 UTC m=+66.212960234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:28.082796 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.082769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:28.137448 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.137416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:28.181487 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.181454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:28.184166 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.184143    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:28.458424 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.458346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:28.483570 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.483545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:28.483719 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:28.483699    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:28.483782 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:28.483764    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.48374794 +0000 UTC m=+38.616070294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:28.561828 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.561799    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4p5d6"]
Oct 31 16:02:28.566026 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:02:28.565993    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda029ca40_3196_4f27_b72a_1d2a2b76197e.slice/crio-f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4 WatchSource:0}: Error finding container f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4: Status 404 returned error can't find the container with id f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4
Oct 31 16:02:28.610268 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.610237    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4p5d6" event={"ID":"a029ca40-3196-4f27-b72a-1d2a2b76197e","Type":"ContainerStarted","Data":"f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4"}
Oct 31 16:02:28.854592 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:28.854505    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5dfz"]
Oct 31 16:02:28.858153 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:02:28.858127    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55552918_3517_4b42_8e9b_b69bbe124e03.slice/crio-be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b WatchSource:0}: Error finding container be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b: Status 404 returned error can't find the container with id be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b
Oct 31 16:02:29.614663 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:29.614626    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5dfz" event={"ID":"55552918-3517-4b42-8e9b-b69bbe124e03","Type":"ContainerStarted","Data":"be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b"}
Oct 31 16:02:30.782999 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.782263    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g4jw4"]
Oct 31 16:02:30.802301 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.802274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.851448 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.851405    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 16:02:30.890653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.890482    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 16:02:30.890653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.890517    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\""
Oct 31 16:02:30.890653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.890537    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 16:02:30.890653 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.890580    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 16:02:30.898916 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.898887    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g4jw4"]
Oct 31 16:02:30.899480 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.899462    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"]
Oct 31 16:02:30.902017 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.901990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.902114 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.902024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.902114 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.902077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.902238 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.902215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv275\" (UniqueName: \"kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.902291 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.902279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:30.917612 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.917592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:30.933662 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.933610    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"]
Oct 31 16:02:30.967017 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.966991    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:30.967017 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.966991    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qmh6f\""
Oct 31 16:02:30.967219 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:30.967100    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 16:02:31.003178 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003310 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003310 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003310 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzcp\" (UniqueName: \"kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp\") pod \"migrator-66df5d496b-2jdw2\" (UID: \"59566ce9-2437-417b-b4cb-150e75e870fe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:31.003461 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv275\" (UniqueName: \"kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003461 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003767 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.003744    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.003862 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.003845    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.003922 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.003909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.50388972 +0000 UTC m=+37.636212073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.004316 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.004292    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.004398 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.004300    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.004398 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.004385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.504369157 +0000 UTC m=+37.636691512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.060373 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.060344    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv275\" (UniqueName: \"kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.104497 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.104462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzcp\" (UniqueName: \"kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp\") pod \"migrator-66df5d496b-2jdw2\" (UID: \"59566ce9-2437-417b-b4cb-150e75e870fe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:31.161567 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.161504    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzcp\" (UniqueName: \"kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp\") pod \"migrator-66df5d496b-2jdw2\" (UID: \"59566ce9-2437-417b-b4cb-150e75e870fe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:31.245410 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.245382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:31.506970 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.506888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.506970 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.506947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:31.507190 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.507040    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.507190 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.507053    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.507190 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.507109    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.507092241 +0000 UTC m=+38.639414591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:31.507190 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:31.507131    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.507117626 +0000 UTC m=+38.639439982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:31.622700 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.622436    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bvb4g" event={"ID":"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183","Type":"ContainerStarted","Data":"52b0891602ea0cda6f2b9b235de79b64e5b52e018cccee63d1b9286735a8eb3f"}
Oct 31 16:02:31.631938 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.631908    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"]
Oct 31 16:02:31.636008 ip-10-0-140-84 kubenswrapper[2580]: W1031 16:02:31.635978    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59566ce9_2437_417b_b4cb_150e75e870fe.slice/crio-39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844 WatchSource:0}: Error finding container 39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844: Status 404 returned error can't find the container with id 39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844
Oct 31 16:02:31.728126 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:31.728073    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bvb4g" podStartSLOduration=2.6493940609999997 podStartE2EDuration="7.728057616s" totalImagesPullingTime="5.078663555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:31.726980356 +0000 UTC m=+37.859302741" watchObservedRunningTime="2026-10-31 16:02:31.728057616 +0000 UTC m=+37.860379989"
Oct 31 16:02:32.515684 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:32.515647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:32.515706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:32.515744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515795    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515854    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515945    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515856    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.515842326 +0000 UTC m=+40.648164696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : configmap "kube-rbac-proxy" not found
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515979    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.515967882 +0000 UTC m=+40.648290232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:32.516122 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:32.515989    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:40.515983841 +0000 UTC m=+46.648306191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:32.626789 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:32.626745    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2" event={"ID":"59566ce9-2437-417b-b4cb-150e75e870fe","Type":"ContainerStarted","Data":"39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844"}
Oct 31 16:02:33.631624 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:33.631568    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5dfz" event={"ID":"55552918-3517-4b42-8e9b-b69bbe124e03","Type":"ContainerStarted","Data":"e09e06bc60f6d2b8ab1879e510e5517d6bb2b59bccabbb9bc597d488c932cd32"}
Oct 31 16:02:33.632180 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:33.631848    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:33.738268 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:33.738218    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5dfz" podStartSLOduration=35.524709279 podStartE2EDuration="39.738203593s" totalImagesPullingTime="4.213494314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:33.73779707 +0000 UTC m=+39.870119465" watchObservedRunningTime="2026-10-31 16:02:33.738203593 +0000 UTC m=+39.870525965"
Oct 31 16:02:34.529272 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.529235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:34.529436 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.529281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:34.532381 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:34.529801    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:34.532381 ip-10-0-140-84 kubenswrapper[2580]: E1031 16:02:34.529886    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.529867526 +0000 UTC m=+44.662189890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:34.532381 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.530196    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:34.635092 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.635064    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2" event={"ID":"59566ce9-2437-417b-b4cb-150e75e870fe","Type":"ContainerStarted","Data":"f37e496c4493c446d1d9b6ed522c8169ee90a8aab5bd26063d6ce19d579296c4"}
Oct 31 16:02:34.741370 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.741327    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2" podStartSLOduration=2.531540631 podStartE2EDuration="4.741314121s" totalImagesPullingTime="2.20977349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:34.740849547 +0000 UTC m=+40.873171921" watchObservedRunningTime="2026-10-31 16:02:34.741314121 +0000 UTC m=+40.873636492"
Oct 31 16:02:34.875302 ip-10-0-140-84 kubenswrapper[2580]: I1031 16:02:34.875103    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:34.875272 ip-10-0-140-84 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 16:02:34.895009 ip-10-0-140-84 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 16:02:34.895178 ip-10-0-140-84 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 16:02:34.895228 ip-10-0-140-84 systemd[1]: kubelet.service: Consumed 2.272s CPU time.
Oct 31 16:02:34.910296 ip-10-0-140-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 16:02:34.966611 ip-10-0-140-84 kubenswrapper[4417]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:34.966611 ip-10-0-140-84 kubenswrapper[4417]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 16:02:34.966611 ip-10-0-140-84 kubenswrapper[4417]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:34.966611 ip-10-0-140-84 kubenswrapper[4417]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 16:02:34.968580 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968513    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.968580 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968581    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968585    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968588    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968591    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968594    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968604    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968609    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968628    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968633    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968636    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968639    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968643    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968646    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968649    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968652    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968655    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968658    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968661    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.968672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968664    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968666    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968669    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968672    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968675    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968678    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968682    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968685    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968688    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968691    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968694    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968696    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968699    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968701    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968704    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968707    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968709    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968712    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968715    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.969130 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968718    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968727    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968730    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968733    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968735    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968738    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968740    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968743    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968745    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968748    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968751    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968754    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968756    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968759    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968761    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968764    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968767    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968769    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968772    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968775    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.969675 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968778    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968781    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968783    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968786    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968788    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968791    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968794    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968796    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968799    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968802    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968805    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968807    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968810    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968813    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968821    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968824    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968826    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968829    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968831    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968834    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.970186 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968836    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968839    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968842    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968844    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968847    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968849    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968852    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968855    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968857    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968860    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968862    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968865    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968867    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968870    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968873    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968875    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968878    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968881    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968883    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968886    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.970700 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968888    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968891    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968894    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968896    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968899    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968901    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968904    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.968914    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969647    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969655    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969658    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969661    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969664    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969666    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969670    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969672    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969675    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969686    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969689    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969691    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.971190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969694    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969697    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969700    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969702    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969705    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969707    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969710    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969713    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969716    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969719    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969722    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969724    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969728    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969732    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969735    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969737    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969740    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969744    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969748    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.971712 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969751    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969755    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969758    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969760    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969766    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969769    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969772    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969775    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969777    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969780    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969782    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969785    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969788    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969790    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969793    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969796    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969798    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969802    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969804    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.972198 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969807    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969810    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969813    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969816    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969818    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969821    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969824    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969826    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969829    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969832    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969834    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969837    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969839    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969842    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969845    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969848    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969850    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969855    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969858    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969860    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.972698 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969863    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969866    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969869    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969871    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969874    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969877    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969880    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969882    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969885    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969887    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969890    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969892    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969895    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969898    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969900    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969903    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969905    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969908    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969910    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969913    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.973192 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969916    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969918    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969921    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969924    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969926    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969929    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969931    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969935    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969938    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969942    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969944    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969947    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969949    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969952    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969955    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.969957    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970411    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970416    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970420    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.973721 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970423    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970426    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970428    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970431    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970434    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970437    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970439    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970442    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970445    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970447    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970450    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970452    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970455    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970458    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970460    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970463    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970465    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970468    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970470    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970473    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.974243 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970475    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970479    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970483    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970486    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970488    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970491    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970494    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970497    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970499    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970502    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970505    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970507    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970510    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970512    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970515    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970517    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970520    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970523    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970525    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970528    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.974755 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970531    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970533    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970536    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970538    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970541    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970543    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970546    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970549    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970551    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970554    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970556    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970559    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970562    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970564    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970568    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970571    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970574    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970576    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970579    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970583    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.975287 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970586    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970589    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970592    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970594    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970597    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970599    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970602    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970605    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970608    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970627    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970631    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970633    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970636    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970639    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970641    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970644    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970647    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970649    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970652    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.975894 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970654    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970657    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970659    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970662    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970664    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970667    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970670    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970673    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970677    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970680    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970683    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970685    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970688    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970692    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970696    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970700    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970703    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970706    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970708    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.976651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970711    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970714    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970717    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970721    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.970724    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970813    4417 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970821    4417 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970826    4417 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970830    4417 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970835    4417 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970838    4417 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970843    4417 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970847    4417 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970850    4417 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970853    4417 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970856    4417 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970859    4417 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970862    4417 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970866    4417 flags.go:64] FLAG: --cgroup-root=""
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970869    4417 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970873    4417 flags.go:64] FLAG: --client-ca-file=""
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970876    4417 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970879    4417 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 16:02:34.977484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970884    4417 flags.go:64] FLAG: --cluster-domain=""
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970886    4417 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970890    4417 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970892    4417 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970896    4417 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970900    4417 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970902    4417 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970905    4417 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970909    4417 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970912    4417 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970915    4417 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970918    4417 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970923    4417 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970926    4417 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970930    4417 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970933    4417 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970936    4417 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970939    4417 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970942    4417 flags.go:64] FLAG: --enable-server="true"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970945    4417 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970949    4417 flags.go:64] FLAG: --event-burst="100"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970952    4417 flags.go:64] FLAG: --event-qps="50"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970955    4417 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970958    4417 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 16:02:34.978549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970961    4417 flags.go:64] FLAG: --eviction-hard=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970965    4417 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970968    4417 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970971    4417 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970973    4417 flags.go:64] FLAG: --eviction-soft=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970977    4417 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970980    4417 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970983    4417 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970987    4417 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970990    4417 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970993    4417 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.970996    4417 flags.go:64] FLAG: --feature-gates=""
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971000    4417 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971003    4417 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971006    4417 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971010    4417 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971013    4417 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971016    4417 flags.go:64] FLAG: --help="false"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971019    4417 flags.go:64] FLAG: --hostname-override="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971022    4417 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971027    4417 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971030    4417 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971034    4417 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 16:02:34.979588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971037    4417 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971040    4417 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971043    4417 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971046    4417 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971049    4417 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971052    4417 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971055    4417 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971058    4417 flags.go:64] FLAG: --kube-reserved=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971061    4417 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971064    4417 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971067    4417 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971070    4417 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971073    4417 flags.go:64] FLAG: --lock-file=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971076    4417 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971079    4417 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971082    4417 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971087    4417 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971090    4417 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971094    4417 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971097    4417 flags.go:64] FLAG: --logging-format="text"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971100    4417 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971103    4417 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971106    4417 flags.go:64] FLAG: --manifest-url=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971109    4417 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971113    4417 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 16:02:34.980591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971117    4417 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971120    4417 flags.go:64] FLAG: --max-pods="110"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971123    4417 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971126    4417 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971130    4417 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971133    4417 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971136    4417 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971139    4417 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971142    4417 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971149    4417 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971153    4417 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971156    4417 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971159    4417 flags.go:64] FLAG: --pod-cidr=""
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971161    4417 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971164    4417 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971167    4417 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971170    4417 flags.go:64] FLAG: --port="10250"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971173    4417 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971176    4417 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00abf68ac65f53838"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971179    4417 flags.go:64] FLAG: --qos-reserved=""
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971182    4417 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971185    4417 flags.go:64] FLAG: --register-node="true"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971188    4417 flags.go:64] FLAG: --register-with-taints=""
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971192    4417 flags.go:64] FLAG: --registry-burst="10"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971194    4417 flags.go:64] FLAG: --registry-qps="5"
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971197    4417 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 16:02:34.981702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971201    4417 flags.go:64] FLAG: --reserved-memory=""
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971205    4417 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971208    4417 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971211    4417 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971213    4417 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971216    4417 flags.go:64] FLAG: --runonce="false"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971219    4417 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971222    4417 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971225    4417 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971228    4417 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971232    4417 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971236    4417 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971239    4417 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971242    4417 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971245    4417 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971248    4417 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971250    4417 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971253    4417 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971257    4417 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971261    4417 flags.go:64] FLAG: --system-cgroups=""
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971264    4417 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971269    4417 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971272    4417 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971275    4417 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971279    4417 flags.go:64] FLAG: --tls-min-version=""
Oct 31 16:02:34.982996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971282    4417 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971285    4417 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971288    4417 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971291    4417 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971294    4417 flags.go:64] FLAG: --v="2"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971298    4417 flags.go:64] FLAG: --version="false"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971302    4417 flags.go:64] FLAG: --vmodule=""
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971306    4417 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971310    4417 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971434    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971437    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971440    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971443    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971445    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971448    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971451    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971453    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971458    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971460    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971463    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971466    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971468    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.983820 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971471    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971473    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971476    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971478    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971482    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971485    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971488    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971490    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971493    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971496    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971498    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971501    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971503    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971506    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971509    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971511    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971514    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971516    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971519    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971522    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.984510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971524    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971527    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971530    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971532    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971535    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971537    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971540    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971544    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971547    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971549    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971552    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971554    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971558    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971562    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971565    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971569    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971574    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971577    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971580    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.985334 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971583    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971586    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971589    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971591    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971594    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971597    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971599    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971602    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971605    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971607    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971610    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971626    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971631    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971642    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971646    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971650    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971654    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971657    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971659    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.986207 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971662    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971666    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971669    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971672    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971674    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971677    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971680    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971682    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971685    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971687    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971691    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971694    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971696    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971699    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971702    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971704    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971707    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971709    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971712    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.986789 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971715    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971717    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971720    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971723    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971726    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971728    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971731    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971734    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971737    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971740    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971742    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971745    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971747    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971750    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971753    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.987295 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971756    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.971781    4417 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971820    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971824    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971827    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971829    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971832    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971835    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971838    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971842    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971844    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971847    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971849    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971852    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971855    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971858    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.987677 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971860    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971863    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971865    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971868    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971870    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971873    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971875    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971878    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971881    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971883    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971886    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971889    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971892    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971894    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971897    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971900    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971902    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971905    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971907    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971910    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.988069 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971912    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971915    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971917    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971920    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971923    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971926    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971929    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971932    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971934    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971937    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971940    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971942    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971945    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971948    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971951    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971953    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971957    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971961    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971964    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971967    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.988552 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971970    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971972    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971975    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971978    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971981    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971983    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971986    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971988    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971991    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971993    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971996    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.971999    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972001    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972004    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972007    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972009    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972013    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972019    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972022    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.989066 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972024    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972027    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972030    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972032    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972035    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972037    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972040    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972042    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972045    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972047    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972050    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972053    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972055    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972058    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972060    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972063    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972065    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972068    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972071    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972073    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.989519 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972076    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972078    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972081    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972084    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972086    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972089    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972092    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972094    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972098    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972102    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972106    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972109    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.972112    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.976296    4417 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.976319    4417 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.976344    4417 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.976351    4417 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976604    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.990032 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976751    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976758    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976765    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976775    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976781    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976786    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976791    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976795    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976800    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976804    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976809    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976813    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976818    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976823    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976827    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976832    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976837    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976841    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976846    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.990484 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976851    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976855    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976860    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976864    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976869    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976873    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976878    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976885    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976889    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976894    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976903    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976909    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976914    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976919    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976923    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976927    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976932    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976936    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976941    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.990976 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976945    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976950    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976954    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976958    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976963    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976967    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976971    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976975    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976980    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976985    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976989    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976993    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.976997    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977005    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977019    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977027    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977032    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977037    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977041    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.991434 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977046    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977051    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977063    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977067    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977072    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977076    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977081    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977086    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977091    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977095    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977100    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977104    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977108    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977112    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977117    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977121    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977126    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977130    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977134    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977139    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.991907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977143    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977149    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977154    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977158    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977163    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977167    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977172    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977176    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977180    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977185    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977189    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977194    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977198    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977202    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977207    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977212    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977216    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977221    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977225    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977230    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.992430 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977235    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977239    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977243    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977248    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977253    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977257    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977262    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977266    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.977311    4417 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977572    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977582    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977596    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977601    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977606    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977611    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977636    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.993045 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977641    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977646    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977651    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977655    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977660    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977664    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977670    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977677    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977688    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977693    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977698    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977702    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977706    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977711    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977715    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977720    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977724    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977728    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977733    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977738    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.993439 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977742    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977747    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977751    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977755    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977760    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977764    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977768    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977773    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977777    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977782    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977786    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977791    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977795    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977800    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977804    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977809    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977813    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977817    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977822    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.993949 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977837    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977841    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977845    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977850    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977854    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977859    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977864    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977868    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977873    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977877    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977902    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977909    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977914    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977918    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977923    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977927    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977931    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977936    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977940    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977944    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.994410 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977948    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977953    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977957    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977961    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977966    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977972    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977976    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977980    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977984    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977989    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977993    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.977997    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978002    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978007    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978011    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978015    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978020    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978024    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978029    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978033    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.994905 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978037    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978041    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978046    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978051    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978055    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978059    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978064    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978068    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978073    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978077    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978081    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978086    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978090    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978094    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978098    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978105    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978113    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978117    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978125    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.995462 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978129    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.978171    4417 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978223    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978229    4417 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978234    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978239    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978245    4417 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978251    4417 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978256    4417 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978261    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978266    4417 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978270    4417 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978274    4417 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978278    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978282    4417 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978286    4417 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 16:02:34.996216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978291    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978296    4417 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978300    4417 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978305    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978309    4417 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978313    4417 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978318    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978322    4417 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978326    4417 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978330    4417 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978335    4417 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978339    4417 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978344    4417 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978348    4417 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978352    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978356    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978362    4417 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978366    4417 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978370    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 16:02:34.996672 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978375    4417 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978379    4417 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978383    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978388    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978392    4417 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978396    4417 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978401    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978405    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978410    4417 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978415    4417 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978419    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978424    4417 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978428    4417 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978433    4417 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978437    4417 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978441    4417 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978445    4417 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978450    4417 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978454    4417 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978458    4417 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 16:02:34.997228 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978462    4417 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978467    4417 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978471    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978475    4417 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978479    4417 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978483    4417 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978488    4417 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978492    4417 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978497    4417 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978501    4417 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978506    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978510    4417 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978515    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978520    4417 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978524    4417 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978528    4417 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978533    4417 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978537    4417 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978541    4417 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978545    4417 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 16:02:34.997914 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978549    4417 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978553    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978557    4417 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978561    4417 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978565    4417 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978569    4417 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978573    4417 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978577    4417 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978582    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978586    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978591    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978595    4417 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978599    4417 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978604    4417 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978608    4417 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978629    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978634    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978638    4417 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978642    4417 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 16:02:34.998418 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978649    4417 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978655    4417 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978660    4417 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978666    4417 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978671    4417 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978681    4417 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978688    4417 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978693    4417 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978697    4417 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978702    4417 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978706    4417 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978711    4417 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978717    4417 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:34.978721    4417 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.978800    4417 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980345    4417 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980418    4417 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980728    4417 server.go:1014] "Starting client certificate rotation"
Oct 31 16:02:34.998925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980881    4417 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980931    4417 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:54 +0000 UTC" deadline="2028-05-07 18:00:24.737153556 +0000 UTC"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.980978    4417 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13297h57m49.756178016s"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.981474    4417 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.983172    4417 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.984505    4417 log.go:25] "Validated CRI v1 runtime API"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.986348    4417 log.go:25] "Validated CRI v1 image API"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.986957    4417 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 16:02:34.999357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.988325    4417 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e0823901-9792-43b1-a0fc-ec3ba05a5aee:/dev/nvme0n1p4 e4ae8f0b-f78f-4cb3-a43a-f9ba54abd905:/dev/nvme0n1p3]
Oct 31 16:02:34.999575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.988345    4417 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4/volumes/kubernetes.io~projected/kube-api-access-btk9z:{mountpoint:/var/lib/kubelet/pods/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4/volumes/kubernetes.io~projected/kube-api-access-btk9z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57/volumes/kubernetes.io~projected/kube-api-access-9v6mn:{mountpoint:/var/lib/kubelet/pods/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57/volumes/kubernetes.io~projected/kube-api-access-9v6mn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~projected/kube-api-access-s466z:{mountpoint:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~projected/kube-api-access-s466z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55552918-3517-4b42-8e9b-b69bbe124e03/volumes/kubernetes.io~projected/kube-api-access-5bf7v:{mountpoint:/var/lib/kubelet/pods/55552918-3517-4b42-8e9b-b69bbe124e03/volumes/kubernetes.io~projected/kube-api-access-5bf7v major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59566ce9-2437-417b-b4cb-150e75e870fe/volumes/kubernetes.io~projected/kube-api-access-cwzcp:{mountpoint:/var/lib/kubelet/pods/59566ce9-2437-417b-b4cb-150e75e870fe/volumes/kubernetes.io~projected/kube-api-access-cwzcp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c5bd675-1035-4a49-af88-1f600d0bd7b7/volumes/kubernetes.io~projected/kube-api-access-7btft:{mountpoint:/var/lib/kubelet/pods/5c5bd675-1035-4a49-af88-1f600d0bd7b7/volumes/kubernetes.io~projected/kube-api-access-7btft major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85669af1-ab91-42af-a131-b3d1c858e737/volumes/kubernetes.io~projected/kube-api-access-jv275:{mountpoint:/var/lib/kubelet/pods/85669af1-ab91-42af-a131-b3d1c858e737/volumes/kubernetes.io~projected/kube-api-access-jv275 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f20fe20-7b1b-4a7f-a855-6520f94f9603/volumes/kubernetes.io~projected/kube-api-access-shb5f:{mountpoint:/var/lib/kubelet/pods/8f20fe20-7b1b-4a7f-a855-6520f94f9603/volumes/kubernetes.io~projected/kube-api-access-shb5f major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea/volumes/kubernetes.io~projected/kube-api-access-jk7d8:{mountpoint:/var/lib/kubelet/pods/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea/volumes/kubernetes.io~projected/kube-api-access-jk7d8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a029ca40-3196-4f27-b72a-1d2a2b76197e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a029ca40-3196-4f27-b72a-1d2a2b76197e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~projected/kube-api-access-ggz98:{mountpoint:/var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~projected/kube-api-access-ggz98 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b04f7600-352a-47ca-8901-98ab6be3dfb2/volumes/kubernetes.io~projected/kube-api-access-z8vcj:{mountpoint:/var/lib/kubelet/pods/b04f7600-352a-47ca-8901-98ab6be3dfb2/volumes/kubernetes.io~projected/kube-api-access-z8vcj major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd4f1c13-63af-4f5b-b0b5-57dab70784af/volumes/kubernetes.io~projected/kube-api-access-cz4xk:{mountpoint:/var/lib/kubelet/pods/cd4f1c13-63af-4f5b-b0b5-57dab70784af/volumes/kubernetes.io~projected/kube-api-access-cz4xk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e201d4ab-e257-44f9-8595-06552d621cdd/volumes/kubernetes.io~projected/kube-api-access-6dwqj:{mountpoint:/var/lib/kubelet/pods/e201d4ab-e257-44f9-8595-06552d621cdd/volumes/kubernetes.io~projected/kube-api-access-6dwqj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~projected/kube-api-access-vqmbm:{mountpoint:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~projected/kube-api-access-vqmbm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0e720fa9633fea72c9fa19c6c8451088f40ca77aa20c1e55aa965a69397b906d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b6364c0266c53530bc6eea15722a3227c6101fe9a80a9dd6a25d625178a664f5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6332e2c5f7ad179f588a1e7b4e7c7da4adee9f5666f1d3b627d41ab469e23fce/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/31330054f8c469e666bfc0d4749ff2dd0bfff3a8c783698a3674e8da064deedc/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/43ffa0cfe1ec9296108edbe280c0fcea0a4ea99aaac4ec218bf3960f65736018/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/860b928961c8acf2c06cc34fca7b546c70e860fafa6afdc923db867d1ed09b4d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/73022540e02c2f7a3ad097a97bb41c603a476e3860ea4d687cf97fbed2c16edc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a2a76f64deb569fd8507f35cc78e759cc2d88fc74e61c5d6afa542703e92576b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/62f56220e1edf91b54500133766fe9debe5f07039c5463a2cbef71321be2a60a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d487b49ef7ef5bb19d23d46db60719fbff68fa52d1f6748cf04c35450820d328/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e4990ec20737a3155400389450b8eb958f4c5bb77eb8f0c862736d3d372363f5/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/9f2fdb5d129770e60b5f8a1a05614ff30fe3993086fd1e67ef7e3313b4a3ba6e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a41f3b2aefaa81d792f4af26e3f1ca157c90af13d2d5fecb73b4806127f4790a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/66734641be9c34d4c34a81115ccbc4edd126b8588cd1f4a0afef7c8ba567ac80/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ba5db19a65a3c62a53b711ca821df04d355eace4c4e996b7aefca927efbb915d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/225eaf1c83096857186614c29af88cc861be97b847e2fae9205730b8886a7fbf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8268969ba64b01b52d157f75e169cdafc7b89c9318142214b6d13c1dc4c4fb55/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ff8df540a076a93088d58cba9150020815326169be1845412606cf1a42dbb6ca/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/06b7a6ce2ce4f5e196783f7a5fcede6c8b6a4e8d7504c3cc8f41fe7896577b3d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6b96ba3fb3b478e66f50caccf2d821580d7d7816ea6aa00b28a7ed656cf84891/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c672d03986f1ec47a9496cb0ffde22df150647a2a317f95875188a795ea2457c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/3c7cb5205a27bec67f3701262a1d6d093021a4aba0612438c755676c45c4fe38/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/2ce4569190be6ccef9abecf1e9cd54f6c274b1626b08ff9f3caeade21bb216e3/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/51180028d013461b92ea088956ff41983e7f03df2100922aefcc13e0890d20a5/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/e53d9692b82cca58e56fe64f6bebe1320c6a8af91ef347b8da24a89bbe9a6275/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/7c9324c262355d2c2156b85fd9b23ece01109318092f300f1f3d735b0a9bc508/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/60b7b05be0ea92dfed37ded51611242473fbc5c2dc8e8ff46d2f68e0742111b1/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/5c326a35dbcd6f3fe179ca5c02c8e2547513314e93650cdfde5cb25f75da5388/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/a56bbd3af93ed720ed7bd0b883664cfbd6e2a55856ca43855d8c69291453cf45/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/7bdc32d85948b02803ae36f83f4f69321e7b657f3a285d2799ec1cf60eb97691/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8b0b727ba4f60222ad45291807b3ee8b39d264db14e2ba83db96e595b07e5122/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/bf17af8b86f0c0beee54cb01012c083380e0e075f92d70f4d5e9991adc9d635d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/aabc22a7846a2671cbdaa449b4b7a83d86673babf90e61fa441b7ab4994197f6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bbd06c12d077dd79dd6d69e6843dff096d3ab1fe21b4ef2596571dd402d9d350/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ec36d0d88d528238bd37b5fcaff752d06700670c929439ea4edc7c1bc167cab8/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/59590c2e54b38754adb8645f47936b13afc5460ce0b9d60d9cb27587888da191/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f5bb193c4b04ae5ebe289e15e79700ac6718cefd399204174e0cf9ad3ceb9b95/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1e0d9d47c3dc8256847b99073103fd05c92066c813305aa51607c587a18c6fac/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f28a1d868e512af780d770d29173547ae8813ea31f19042ec1ee1a7488bdaa42/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.994863    4417 manager.go:217] Machine: {Timestamp:2026-10-31 16:02:34.994392241 +0000 UTC m=+0.055967418 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101636 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c15c76a26d340b0f17ae030400b44 SystemUUID:ec2c15c7-6a26-d340-b0f1-7ae030400b44 BootID:bda0d8b5-1ec5-4ee3-9b72-161a11cad89f Filesystems:[{Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~projected/kube-api-access-ggz98 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e201d4ab-e257-44f9-8595-06552d621cdd/volumes/kubernetes.io~projected/kube-api-access-6dwqj DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57/volumes/kubernetes.io~projected/kube-api-access-9v6mn DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e05b5ada621d9221774c0615b0cc32f143ba83d56fb47c27ffc5132a57b78202/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59566ce9-2437-417b-b4cb-150e75e870fe/volumes/kubernetes.io~projected/kube-api-access-cwzcp DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39f88df84e685853cf7a0b81bbb01ca3f03023157753f03da23f752452ddd844/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volumes/kubernetes.io~projected/kube-api-access-vqmbm DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d2ddde25e2420acd5e9b55bba7436f41498a65d5aedb9aab98ae2313ece351c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/009e5e6db963f651600a0bfcf982f80ee511101b29fddb6abec06c9004aaa3bd/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~projected/kube-api-access-s466z DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb58743c-05b8-4d94-b8d0-84a80eb9c06b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/55552918-3517-4b42-8e9b-b69bbe124e03/volumes/kubernetes.io~projected/kube-api-access-5bf7v DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f78147d4569395fc850a0995112e2d29ada9f93e5843d8aaabfac60d457e15c4/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be426f4e4a90cd25b649a617e06e9c51a3ca48d780c041ed70f20261ea4eaa4b/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/85669af1-ab91-42af-a131-b3d1c858e737/volumes/kubernetes.io~projected/kube-api-access-jv275 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd4f1c13-63af-4f5b-b0b5-57dab70784af/volumes/kubernetes.io~projected/kube-api-access-cz4xk DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0157b212797e081791fd43a354dc984d73498066628ba0ee79abfe139567aa07/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbd2b662851ce6c55fa77f33d23e7a02196e650f4b47fb3551bad2fea60ca688/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad99b54ad76a53d28e7861d64cea4c29d70b87c6d83dbf735e9c684d014b9382/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98719cf97cf93efdc7a939c8f75e131bc635d5bfad9e0fa316f99ad4fdd7d90c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a6e8173b18a25d8e2193c970cce0955643aaea2f19b8d9494afb912f0a116cd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b04f7600-352a-47ca-8901-98ab6be3dfb2/volumes/kubernetes.io~projected/kube-api-access-z8vcj DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a029ca40-3196-4f27-b72a-1d2a2b76197e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4c7fd9a4-5924-4e14-b744-d9b11ea3f696/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c5bd675-1035-4a49-af88-1f600d0bd7b7/volumes/kubernetes.io~projected/kube-api-access-7btft DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4/volumes/kubernetes.io~projected/kube-api-access-btk9z DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f20fe20-7b1b-4a7f-a855-6520f94f9603/volumes/kubernetes.io~projected/kube-api-access-shb5f DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12ae3981883a9d19ba275cabd2b1a2be1d062f9deb833064cb98eae949713c4e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea/volumes/kubernetes.io~projected/kube-api-access-jk7d8 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:009e5e6db963f65 MacAddress:0a:5b:2e:14:c6:0f Speed:10000 Mtu:8901} {Name:39f88df84e68585 MacAddress:2a:e8:94:72:26:98 Speed:10000 Mtu:8901} {Name:be426f4e4a90cd2 MacAddress:ae:6b:f2:09:a5:49 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:df:f2:12:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:b8:15:ce:51:8a Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:df:f2:12:7d Speed:0 Mtu:9001} {Name:f78147d4569395f MacAddress:4e:9f:03:14:df:4d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:65:bf:3e:71:e6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:f7:66:92:7e:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.995093    4417 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.995131    4417 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.995663    4417 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.995689    4417 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-84.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996073    4417 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996087    4417 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996101    4417 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996126    4417 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996236    4417 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996762    4417 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996888    4417 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996922    4417 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996935    4417 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996945    4417 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996953    4417 kubelet.go:405] "Adding apiserver pod source"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.996961    4417 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.997463    4417 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.997473    4417 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.997893    4417 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.999674    4417 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.999683    4417 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 16:02:34.999800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.999765    4417 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.999933    4417 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:34.999960    4417 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000041    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000052    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000058    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000063    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000069    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000075    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000082    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000087    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000094    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000100    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000114    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000123    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000149    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 16:02:35.000495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.000155    4417 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 16:02:35.003329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.003315    4417 server.go:1263] "Started kubelet"
Oct 31 16:02:35.003421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.003395    4417 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 16:02:35.003554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.003411    4417 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 16:02:35.004390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.003572    4417 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 16:02:35.003848 ip-10-0-140-84 systemd[1]: Started Kubernetes Kubelet.
Oct 31 16:02:35.004909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.004563    4417 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 16:02:35.006002 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.005815    4417 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 16:02:35.006002 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.005974    4417 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 16:02:35.007500 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.006934    4417 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:02:35.007605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007548    4417 factory.go:55] Registering systemd factory
Oct 31 16:02:35.007605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007565    4417 factory.go:223] Registration of the systemd container factory successfully
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007907    4417 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007951    4417 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007993    4417 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:56 +0000 UTC" deadline="2028-04-24 15:10:17.308578118 +0000 UTC"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008025    4417 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12983h7m42.300556287s"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.007796    4417 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008137    4417 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008354    4417 factory.go:153] Registering CRI-O factory
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008370    4417 factory.go:223] Registration of the crio container factory successfully
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008445    4417 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008464    4417 factory.go:103] Registering Raw factory
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.008479    4417 manager.go:1199] Started watching for new ooms in manager
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.009281    4417 manager.go:319] Starting recovery of all containers
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011210    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7fd9a4-5924-4e14-b744-d9b11ea3f696" volumeName="kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011269    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7fd9a4-5924-4e14-b744-d9b11ea3f696" volumeName="kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011302    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c5bd675-1035-4a49-af88-1f600d0bd7b7" volumeName="kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011320    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e4b8e3-1495-49dd-932a-0bbc5c6b7183" volumeName="kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011341    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b04f7600-352a-47ca-8901-98ab6be3dfb2" volumeName="kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011358    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29a5a550-ba91-4cd4-92a7-bb61db0e3eb5" volumeName="kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011388    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44ec4a55-0dd7-41f6-8837-eac4e3d5cb57" volumeName="kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011427    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c7fd9a4-5924-4e14-b744-d9b11ea3f696" volumeName="kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011447    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94bfbeb1-ecc8-479f-a02e-c6b1d55daaea" volumeName="kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011468    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb58743c-05b8-4d94-b8d0-84a80eb9c06b" volumeName="kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011511    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb58743c-05b8-4d94-b8d0-84a80eb9c06b" volumeName="kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011522    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55552918-3517-4b42-8e9b-b69bbe124e03" volumeName="kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011539    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59566ce9-2437-417b-b4cb-150e75e870fe" volumeName="kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011560    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb58743c-05b8-4d94-b8d0-84a80eb9c06b" volumeName="kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011580    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb58743c-05b8-4d94-b8d0-84a80eb9c06b" volumeName="kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011592    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94bfbeb1-ecc8-479f-a02e-c6b1d55daaea" volumeName="kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011651    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e4b8e3-1495-49dd-932a-0bbc5c6b7183" volumeName="kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011684    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b04f7600-352a-47ca-8901-98ab6be3dfb2" volumeName="kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011696    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29a5a550-ba91-4cd4-92a7-bb61db0e3eb5" volumeName="kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011736    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f20fe20-7b1b-4a7f-a855-6520f94f9603" volumeName="kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011750    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4f1c13-63af-4f5b-b0b5-57dab70784af" volumeName="kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011763    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4" volumeName="kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011791    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44ec4a55-0dd7-41f6-8837-eac4e3d5cb57" volumeName="kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011803    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85669af1-ab91-42af-a131-b3d1c858e737" volumeName="kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011820    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85669af1-ab91-42af-a131-b3d1c858e737" volumeName="kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011866    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a029ca40-3196-4f27-b72a-1d2a2b76197e" volumeName="kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011912    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b04f7600-352a-47ca-8901-98ab6be3dfb2" volumeName="kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011948    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4f1c13-63af-4f5b-b0b5-57dab70784af" volumeName="kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011983    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4f1c13-63af-4f5b-b0b5-57dab70784af" volumeName="kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.011994    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c5bd675-1035-4a49-af88-1f600d0bd7b7" volumeName="kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012009    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85669af1-ab91-42af-a131-b3d1c858e737" volumeName="kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012035    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f20fe20-7b1b-4a7f-a855-6520f94f9603" volumeName="kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012049    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f20fe20-7b1b-4a7f-a855-6520f94f9603" volumeName="kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012082    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e201d4ab-e257-44f9-8595-06552d621cdd" volumeName="kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012141    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb58743c-05b8-4d94-b8d0-84a80eb9c06b" volumeName="kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012156    4417 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd4f1c13-63af-4f5b-b0b5-57dab70784af" volumeName="kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk" seLinuxMountContext=""
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012167    4417 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 16:02:35.012346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.012175    4417 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 16:02:35.039452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.039380    4417 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 16:02:35.041179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.041151    4417 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 16:02:35.041179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.041173    4417 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 16:02:35.041345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.041193    4417 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 16:02:35.041345 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.041235    4417 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 16:02:35.056944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056919    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="a9ade5cc51d64bd151389a2b10b6381e2acd3dd4e88a34e52e698c8e0f6f267e" exitCode=0
Oct 31 16:02:35.056944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056941    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="b68de7e1e5abe05032ca3d19fd20585d88601089cbfd1a861e54d1e9314aee85" exitCode=0
Oct 31 16:02:35.056944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056947    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="b21e50682622680997a1d3800c42ae9537f30fb034992ba199d1de825cdbcd01" exitCode=0
Oct 31 16:02:35.057113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056953    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="aae147eadb04cf107178d52bb17b2ad13474a3d880a1798fe1bb06f585f12114" exitCode=0
Oct 31 16:02:35.057113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056958    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="cc285be1a9deee4b33823082c44648d697047b39442ebbbfc01bfc42caa82c71" exitCode=0
Oct 31 16:02:35.057113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.056963    4417 generic.go:356] "Generic (PLEG): container finished" podID="cd4f1c13-63af-4f5b-b0b5-57dab70784af" containerID="98926a40b292700ef0aac37f7a67eb12e714229ac64f8ef372913aaebf2954b5" exitCode=0
Oct 31 16:02:35.074642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.074608    4417 manager.go:324] Recovery completed
Oct 31 16:02:35.074642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.074632    4417 generic.go:356] "Generic (PLEG): container finished" podID="b65df5c6b29af4cc29df4b46ea700007" containerID="e9b57f3ec02ee4891218672c519f710b96fc898b3c49d1eabfc352bbd373cf21" exitCode=0
Oct 31 16:02:35.086402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.086388    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.087611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.087592    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.087693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.087639    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.087693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.087655    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.088881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.088862    4417 cpu_manager.go:226] "Starting" policy="none"
Oct 31 16:02:35.088881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.088880    4417 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 16:02:35.088973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.088880    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.088973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.088902    4417 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 16:02:35.089075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089060    4417 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 16:02:35.089111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089075    4417 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 16:02:35.089111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089092    4417 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 16:02:35.089111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089097    4417 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 16:02:35.089111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089103    4417 policy_none.go:50] "Start"
Oct 31 16:02:35.089111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089111    4417 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 16:02:35.089284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089121    4417 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 16:02:35.089284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089248    4417 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 16:02:35.089284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089255    4417 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 16:02:35.089284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.089260    4417 policy_none.go:44] "Start"
Oct 31 16:02:35.090433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.090411    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.090563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.090449    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.090563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.090470    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.094685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.094657    4417 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 16:02:35.097619 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.097605    4417 manager.go:341] "Starting Device Plugin manager"
Oct 31 16:02:35.097711 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.097648    4417 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 16:02:35.097711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.097659    4417 server.go:87] "Starting device plugin registration server"
Oct 31 16:02:35.097957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.097945    4417 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 16:02:35.098013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.097959    4417 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 16:02:35.098098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.098081    4417 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 16:02:35.098211 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.098180    4417 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 16:02:35.098292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.098219    4417 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 16:02:35.098292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.098225    4417 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 16:02:35.098398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.098286    4417 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:35.098209737Z","UUID":"c8f2a584-426e-4399-8452-d968b027c9e0","Handler":null,"Name":"","Endpoint":""}
Oct 31 16:02:35.099411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.099392    4417 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 16:02:35.099483 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.099419    4417 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 16:02:35.099975 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.099946    4417 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 16:02:35.100057 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.099990    4417 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-84.us-west-2.compute.internal\" not found"
Oct 31 16:02:35.142135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.141591    4417 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"]
Oct 31 16:02:35.142247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.142183    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.143149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.143133    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.143228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.143162    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.143228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.143179    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.143420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.143408    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.143572 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.143556    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.144189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144171    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.144259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144200    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.144259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144210    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.144259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144171    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.144363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144282    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.144363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144295    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.144447 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.144435    4417 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.144480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144429    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"102c36941ddb017d6ed417f277ac7f2a","Type":"ContainerStarted","Data":"b6eb9ea0adaa255b2c741e8db7054a42d7c9c03dc189ff4bac9937037728065a"}
Oct 31 16:02:35.144480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144471    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"102c36941ddb017d6ed417f277ac7f2a","Type":"ContainerStarted","Data":"a1583d1b9d19b7a0c5f41131dd2974c591403691314b732c5915ab6f5b971dc9"}
Oct 31 16:02:35.144544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144500    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerStarted","Data":"25b3f6d03726bfcdfc5fad2534cd3cc1d623e460808381b204e59e426a3d9460"}
Oct 31 16:02:35.144544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144518    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerDied","Data":"e9b57f3ec02ee4891218672c519f710b96fc898b3c49d1eabfc352bbd373cf21"}
Oct 31 16:02:35.144544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144530    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.144544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.144539    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal" event={"ID":"b65df5c6b29af4cc29df4b46ea700007","Type":"ContainerStarted","Data":"50a4083ae9dcc378ea6dc5ad82c7a0c70f440392421adf943df77b41469e70a2"}
Oct 31 16:02:35.145249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.145236    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.145287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.145260    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.145287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.145270    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.145399 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.145387    4417 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-84.us-west-2.compute.internal\" not found" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.198117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.198094    4417 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 16:02:35.199598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.199580    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:02:35.199718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.199610    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:02:35.199718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.199637    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:02:35.199718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.199658    4417 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.203254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.203234    4417 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:35.203323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.203262    4417 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:35.203363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.203341    4417 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 16:02:35.244886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.244862    4417 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:35.249986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.249966    4417 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:02:35.300671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.300649    4417 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.300739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.300703    4417 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.301197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.301183    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeNotReady"
Oct 31 16:02:35.301235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.301211    4417 setters.go:546] "Node became not ready" node="ip-10-0-140-84.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T16:02:35Z","lastTransitionTime":"2026-10-31T16:02:35Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 16:02:35.304371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.304353    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.304420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.304380    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.304420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.304401    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.307469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.307456    4417 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.339818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.339794    4417 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-84.us-west-2.compute.internal" is owned by different node (oldNodeID="3cc25456-c31f-424c-829b-bfb3671a26c3", newNodeID=""), cleaning up...
Oct 31 16:02:35.364305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.364285    4417 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:35.364378 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.364323    4417 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.364378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.364334    4417 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404805    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404837    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404862    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404901    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404910    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b65df5c6b29af4cc29df4b46ea700007-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"b65df5c6b29af4cc29df4b46ea700007\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.404967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.404911    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102c36941ddb017d6ed417f277ac7f2a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" (UID: \"102c36941ddb017d6ed417f277ac7f2a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.422498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.422474    4417 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:35.422594 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:35.422521    4417 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:35.998112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:35.998086    4417 apiserver.go:52] "Watching apiserver"
Oct 31 16:02:36.077000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.076977    4417 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:36.077133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.077119    4417 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:36.133776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.133750    4417 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:36.133891 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.133788    4417 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:36.133891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.133751    4417 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 16:02:36.134148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.134113    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-kwgkk","openshift-network-diagnostics/network-check-target-g5dfz","openshift-network-operator/iptables-alerter-shkgz","openshift-ovn-kubernetes/ovnkube-node-bmvzx","openshift-multus/multus-additional-cni-plugins-95f9v","openshift-insights/insights-runtime-extractor-g4jw4","openshift-multus/network-metrics-daemon-7q6q2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal","openshift-dns/node-resolver-9d2bp","kube-system/global-pull-secret-syncer-4p5d6","openshift-cluster-node-tuning-operator/tuned-8hdhf","openshift-dns/dns-default-xwx8c","openshift-image-registry/node-ca-jjdf9","openshift-ingress-canary/ingress-canary-bvb4g","openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq","kube-system/konnectivity-agent-n76gt"]
Oct 31 16:02:36.134383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.134367    4417 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 16:02:36.134453 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.134397    4417 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-84.us-west-2.compute.internal"
Oct 31 16:02:36.136303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.136283    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.138433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.138417    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.139434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.139417    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.184105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.184086    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\""
Oct 31 16:02:36.184105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.184088    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\""
Oct 31 16:02:36.185507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185493    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.185643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185609    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 16:02:36.185643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185629    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.185643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185641    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185963    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185993    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.185996    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186034    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186042    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186080    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.186123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186097    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186195    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186208    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186228    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186232    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186244    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186266    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186267    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186268    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186286    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.186484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186379    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 16:02:36.186878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.186866    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 16:02:36.204400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.204382    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.204496 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.204484    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.204545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.204535    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.204697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.204670    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.204932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.204918    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 16:02:36.205071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205057    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.205148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205088    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.205148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205090    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 16:02:36.205148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205099    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 16:02:36.205266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205173    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.205266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205196    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.205327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.205292    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 16:02:36.209326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209299    4417 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 16:02:36.209454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209441    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s466z\" (UniqueName: \"kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.209546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209534    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.209580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209554    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.209608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209583    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.209655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209646    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.209704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209666    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.209704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209689    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.209762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209704    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.209762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209741    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.209834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209761    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.209834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209778    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.209834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209811    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.209834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209829    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209846    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209860    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209883    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209909    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209965    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209989    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.209990    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04f7600-352a-47ca-8901-98ab6be3dfb2-tmp-dir\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.210111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210038    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210116    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210204    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-env-overrides\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210205    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210304    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggz98\" (UniqueName: \"kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210342    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.210402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210399    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210411    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210418    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85669af1-ab91-42af-a131-b3d1c858e737-data-volume\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210430    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210446    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210480    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210505    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210524    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210545    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210558    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210576    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210596    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210680    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210752    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btk9z\" (UniqueName: \"kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210801    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210860    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210901    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210920    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.210985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.210938    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211021    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211044    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211062    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211075    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-daemon-config\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211076    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cni-binary-copy\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211102    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211171    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211190    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211212    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4xk\" (UniqueName: \"kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211229    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwqj\" (UniqueName: \"kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211252    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211284    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-tmp-dir\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.211424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211304    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.211693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211636    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211646    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-tmp\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211696    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shb5f\" (UniqueName: \"kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.211786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211748    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211785    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211824    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.211956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211871    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.212000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.211987    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-tuned\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.212045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212023    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv275\" (UniqueName: \"kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.212102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212063    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:36.212148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212097    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.212148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212131    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:36.212239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212156    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.212239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212188    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.212239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212218    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.212374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212250    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.212374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212283    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.212374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212333    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.212579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212560    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-serviceca\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.212649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212631    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.212695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212668    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.212738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212700    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:36.212738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212731    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7d8\" (UniqueName: \"kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.212821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212762    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.212821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212791    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-script-lib\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.212908 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212860    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.212908 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212902    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.212992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212934    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.212992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212971    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6mn\" (UniqueName: \"kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.213118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.212999    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.213118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213030    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.213118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213064    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.213118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213096    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.213284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213126    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.213284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213153    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.213284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213268    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.213407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213301    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.213407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213337    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vcj\" (UniqueName: \"kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.213407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213366    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.213407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213392    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzcp\" (UniqueName: \"kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp\") pod \"migrator-66df5d496b-2jdw2\" (UID: \"59566ce9-2437-417b-b4cb-150e75e870fe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:36.213575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213422    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.213575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213453    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.213575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213482    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.213575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213512    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.213575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213558    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.213809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.213587    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.214076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.214049    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-konnectivity-ca\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:36.214261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.214244    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.214526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.214509    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b04f7600-352a-47ca-8901-98ab6be3dfb2-config-volume\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.215121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.215096    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85669af1-ab91-42af-a131-b3d1c858e737-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.215218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.215139    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cni-binary-copy\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.215279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.215260    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c5bd675-1035-4a49-af88-1f600d0bd7b7-iptables-alerter-script\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.215433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.215419    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovnkube-config\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.260661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.260582    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s466z\" (UniqueName: \"kubernetes.io/projected/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-kube-api-access-s466z\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.261321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.261294    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-kube-api-access-vqmbm\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.261634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.261603    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btk9z\" (UniqueName: \"kubernetes.io/projected/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-kube-api-access-btk9z\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.261734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.261704    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggz98\" (UniqueName: \"kubernetes.io/projected/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-kube-api-access-ggz98\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:36.262336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.262314    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwqj\" (UniqueName: \"kubernetes.io/projected/e201d4ab-e257-44f9-8595-06552d621cdd-kube-api-access-6dwqj\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.262412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.262340    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4xk\" (UniqueName: \"kubernetes.io/projected/cd4f1c13-63af-4f5b-b0b5-57dab70784af-kube-api-access-cz4xk\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.262549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.262522    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv275\" (UniqueName: \"kubernetes.io/projected/85669af1-ab91-42af-a131-b3d1c858e737-kube-api-access-jv275\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.262862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.262846    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf7v\" (UniqueName: \"kubernetes.io/projected/55552918-3517-4b42-8e9b-b69bbe124e03-kube-api-access-5bf7v\") pod \"network-check-target-g5dfz\" (UID: \"55552918-3517-4b42-8e9b-b69bbe124e03\") " pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:36.263058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.263042    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shb5f\" (UniqueName: \"kubernetes.io/projected/8f20fe20-7b1b-4a7f-a855-6520f94f9603-kube-api-access-shb5f\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.264420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.264404    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btft\" (UniqueName: \"kubernetes.io/projected/5c5bd675-1035-4a49-af88-1f600d0bd7b7-kube-api-access-7btft\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.265354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.265333    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vcj\" (UniqueName: \"kubernetes.io/projected/b04f7600-352a-47ca-8901-98ab6be3dfb2-kube-api-access-z8vcj\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.265475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.265459    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6mn\" (UniqueName: \"kubernetes.io/projected/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-kube-api-access-9v6mn\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.266100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.266085    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7d8\" (UniqueName: \"kubernetes.io/projected/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-kube-api-access-jk7d8\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.266139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.266089    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzcp\" (UniqueName: \"kubernetes.io/projected/59566ce9-2437-417b-b4cb-150e75e870fe-kube-api-access-cwzcp\") pod \"migrator-66df5d496b-2jdw2\" (UID: \"59566ce9-2437-417b-b4cb-150e75e870fe\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2jdw2"
Oct 31 16:02:36.283980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.283957    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 16:02:36.284413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284392    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\""
Oct 31 16:02:36.284537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284518    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e4b8e3-1495-49dd-932a-0bbc5c6b7183-cert\") pod \"ingress-canary-bvb4g\" (UID: \"a1e4b8e3-1495-49dd-932a-0bbc5c6b7183\") " pod="openshift-ingress-canary/ingress-canary-bvb4g"
Oct 31 16:02:36.284684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284664    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 16:02:36.284765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284745    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\""
Oct 31 16:02:36.284765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284751    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qmh6f\""
Oct 31 16:02:36.284897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284854    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 16:02:36.284897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284878    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 16:02:36.284897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284887    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284889    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284937    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.284965    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285028    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285044    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285049    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285076    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\""
Oct 31 16:02:36.285081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285078    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285121    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285181    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285216    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285229    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285308    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 16:02:36.285442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.285360    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\""
Oct 31 16:02:36.291213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.291199    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-ovn-node-metrics-cert\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.292405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.292390    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29a5a550-ba91-4cd4-92a7-bb61db0e3eb5-agent-certs\") pod \"konnectivity-agent-n76gt\" (UID: \"29a5a550-ba91-4cd4-92a7-bb61db0e3eb5\") " pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:36.295638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.295607    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a029ca40-3196-4f27-b72a-1d2a2b76197e-original-pull-secret\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.314169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314149    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314174    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314192    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.314231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314209    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314248    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-hostroot\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314270    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-sys\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314270    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-var-lib-kubelet\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314295    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314322    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314338    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314339    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-multus-certs\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314352    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314297    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314374    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314396    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314406    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-systemd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.314420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314412    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-conf-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314427    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314431    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-etc-kubernetes\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314452    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314458    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysconfig\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314476    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314499    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-bin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314507    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-modprobe-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314499    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.314527    4417 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314547    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314571    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314573    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-netns\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.314596    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:36.814578669 +0000 UTC m=+1.876153850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314607    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-d\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314650    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-dbus\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314651    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314676    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314686    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44ec4a55-0dd7-41f6-8837-eac4e3d5cb57-hosts-file\") pod \"node-resolver-9d2bp\" (UID: \"44ec4a55-0dd7-41f6-8837-eac4e3d5cb57\") " pod="openshift-dns/node-resolver-9d2bp"
Oct 31 16:02:36.314682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314691    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314713    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314648    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-sysctl-conf\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314736    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94bfbeb1-ecc8-479f-a02e-c6b1d55daaea-host\") pod \"node-ca-jjdf9\" (UID: \"94bfbeb1-ecc8-479f-a02e-c6b1d55daaea\") " pod="openshift-image-registry/node-ca-jjdf9"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314741    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314767    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314784    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314787    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-socket-dir-parent\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314772    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-node-log\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314801    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314813    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-k8s-cni-cncf-io\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314819    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-cni-multus\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314842    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314871    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.314889    4417 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314925    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.314935    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:36.814924101 +0000 UTC m=+1.876499264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314925    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-var-lib-kubelet\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314888    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-kubernetes\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314962    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314977    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.314997    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315013    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-systemd-units\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315038    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-slash\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315014    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315049    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-etc-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315056    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a029ca40-3196-4f27-b72a-1d2a2b76197e-kubelet-config\") pod \"global-pull-secret-syncer-4p5d6\" (UID: \"a029ca40-3196-4f27-b72a-1d2a2b76197e\") " pod="kube-system/global-pull-secret-syncer-4p5d6"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315080    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315101    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315120    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315135    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315138    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-log-socket\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315161    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-etc-systemd\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315187    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.315189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315192    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315212    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-system-cni-dir\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315218    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315227    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-lib-modules\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315241    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315275    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-host\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315282    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-system-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315300    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315315    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315329    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315345    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315360    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315384    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-socket-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315387    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-sys-fs\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315391    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315407    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315414    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-host-run-netns\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315428    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315446    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315457    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-run-ovn\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315464    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.315472    4417 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315477    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-bin\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315495    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315500    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd4f1c13-63af-4f5b-b0b5-57dab70784af-cnibin\") pod \"multus-additional-cni-plugins-95f9v\" (UID: \"cd4f1c13-63af-4f5b-b0b5-57dab70784af\") " pod="openshift-multus/multus-additional-cni-plugins-95f9v"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315510    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315480    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85669af1-ab91-42af-a131-b3d1c858e737-crio-socket\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.315520    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:36.815506838 +0000 UTC m=+1.877082008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315542    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315558    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315572    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315586    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315596    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-multus-cni-dir\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315600    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315636    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315648    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c5bd675-1035-4a49-af88-1f600d0bd7b7-host-slash\") pod \"iptables-alerter-shkgz\" (UID: \"5c5bd675-1035-4a49-af88-1f600d0bd7b7\") " pod="openshift-network-operator/iptables-alerter-shkgz"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315647    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-registration-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315662    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-device-dir\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315673    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-kubelet\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315653    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315689    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-var-lib-openvswitch\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315704    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315710    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e201d4ab-e257-44f9-8595-06552d621cdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-lklhq\" (UID: \"e201d4ab-e257-44f9-8595-06552d621cdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lklhq"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315720    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315735    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315742    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb58743c-05b8-4d94-b8d0-84a80eb9c06b-host-cni-netd\") pod \"ovnkube-node-bmvzx\" (UID: \"eb58743c-05b8-4d94-b8d0-84a80eb9c06b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315770    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f20fe20-7b1b-4a7f-a855-6520f94f9603-cnibin\") pod \"multus-kwgkk\" (UID: \"8f20fe20-7b1b-4a7f-a855-6520f94f9603\") " pod="openshift-multus/multus-kwgkk"
Oct 31 16:02:36.316071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.315776    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7fd9a4-5924-4e14-b744-d9b11ea3f696-run\") pod \"tuned-8hdhf\" (UID: \"4c7fd9a4-5924-4e14-b744-d9b11ea3f696\") " pod="openshift-cluster-node-tuning-operator/tuned-8hdhf"
Oct 31 16:02:36.819980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.819947    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:36.820138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.819992    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:36.820138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:36.820028    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:36.820138 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820101    4417 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:36.820138 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820116    4417 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:36.820283 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820101    4417 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:36.820283 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820166    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.820149194 +0000 UTC m=+2.881724371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:36.820283 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820181    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.820172867 +0000 UTC m=+2.881748034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:36.820283 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:36.820195    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.820188304 +0000 UTC m=+2.881763483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:37.829203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:37.829168    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:37.829203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:37.829212    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:37.829239    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829331    4417 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829380    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.829368576 +0000 UTC m=+4.890943739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829331    4417 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829410    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.829404704 +0000 UTC m=+4.890979868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829331    4417 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:37.829607 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:37.829432    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:39.829427307 +0000 UTC m=+4.891002470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:38.839878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:38.839846    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9d2bp_44ec4a55-0dd7-41f6-8837-eac4e3d5cb57/dns-node-resolver/0.log"
Oct 31 16:02:39.840112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:39.840063    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:39.840125    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:39.840167    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840220    4417 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840258    4417 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840220    4417 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840296    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:43.840277251 +0000 UTC m=+8.901852503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840314    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls podName:b04f7600-352a-47ca-8901-98ab6be3dfb2 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:43.840305508 +0000 UTC m=+8.901880675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls") pod "dns-default-xwx8c" (UID: "b04f7600-352a-47ca-8901-98ab6be3dfb2") : secret "dns-default-metrics-tls" not found
Oct 31 16:02:39.840571 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:39.840328    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls podName:85669af1-ab91-42af-a131-b3d1c858e737 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:43.840319379 +0000 UTC m=+8.901894545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g4jw4" (UID: "85669af1-ab91-42af-a131-b3d1c858e737") : secret "insights-runtime-extractor-tls" not found
Oct 31 16:02:40.039680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:40.039647    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jjdf9_94bfbeb1-ecc8-479f-a02e-c6b1d55daaea/node-ca/0.log"
Oct 31 16:02:40.640568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:40.640540    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bvb4g_a1e4b8e3-1495-49dd-932a-0bbc5c6b7183/serve-healthcheck-canary/0.log"
Oct 31 16:02:41.239741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:41.239709    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2jdw2_59566ce9-2437-417b-b4cb-150e75e870fe/migrator/0.log"
Oct 31 16:02:43.863532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.863491    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:43.863532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.863535    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:43.864078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.863563    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:43.864078 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:43.863716    4417 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 16:02:43.864078 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:02:43.863782    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs podName:24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:51.863767235 +0000 UTC m=+16.925342398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs") pod "network-metrics-daemon-7q6q2" (UID: "24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4") : secret "metrics-daemon-secret" not found
Oct 31 16:02:43.864078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.863964    4417 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 16:02:43.866023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.865995    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b04f7600-352a-47ca-8901-98ab6be3dfb2-metrics-tls\") pod \"dns-default-xwx8c\" (UID: \"b04f7600-352a-47ca-8901-98ab6be3dfb2\") " pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:43.866141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.866030    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85669af1-ab91-42af-a131-b3d1c858e737-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g4jw4\" (UID: \"85669af1-ab91-42af-a131-b3d1c858e737\") " pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:43.938953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.938930    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:43.940019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:43.940000    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g4jw4"
Oct 31 16:02:44.304392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:44.304360    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g4jw4"]
Oct 31 16:02:44.345333 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:44.345304    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85669af1_ab91_42af_a131_b3d1c858e737.slice/crio-af602e40d7ac5c03ad02ee4c8d54df652b211b932f75240625317a253a9280fa WatchSource:0}: Error finding container af602e40d7ac5c03ad02ee4c8d54df652b211b932f75240625317a253a9280fa: Status 404 returned error can't find the container with id af602e40d7ac5c03ad02ee4c8d54df652b211b932f75240625317a253a9280fa
Oct 31 16:02:44.350823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:44.350802    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xwx8c"]
Oct 31 16:02:44.355268 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:44.355212    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04f7600_352a_47ca_8901_98ab6be3dfb2.slice/crio-30a79cda3baab5d31a71c573c57a3eaab159eb04f90e4b8a2e3c427d5cd73569 WatchSource:0}: Error finding container 30a79cda3baab5d31a71c573c57a3eaab159eb04f90e4b8a2e3c427d5cd73569: Status 404 returned error can't find the container with id 30a79cda3baab5d31a71c573c57a3eaab159eb04f90e4b8a2e3c427d5cd73569
Oct 31 16:02:44.357822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:44.357802    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:02:44.675853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:44.675784    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:45.098632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.098588    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g4jw4" event={"ID":"85669af1-ab91-42af-a131-b3d1c858e737","Type":"ContainerStarted","Data":"8ce0792ef0dd100c648f00a6b9334872049cad63b0c6b1aadf2c9cbb5e218ad0"}
Oct 31 16:02:45.099057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.098657    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g4jw4" event={"ID":"85669af1-ab91-42af-a131-b3d1c858e737","Type":"ContainerStarted","Data":"af602e40d7ac5c03ad02ee4c8d54df652b211b932f75240625317a253a9280fa"}
Oct 31 16:02:45.099767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.099742    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwx8c" event={"ID":"b04f7600-352a-47ca-8901-98ab6be3dfb2","Type":"ContainerStarted","Data":"30a79cda3baab5d31a71c573c57a3eaab159eb04f90e4b8a2e3c427d5cd73569"}
Oct 31 16:02:45.428283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.428215    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:02:45.428432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.428416    4417 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 16:02:45.445973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.445949    4417 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-84.us-west-2.compute.internal" event="NodeReady"
Oct 31 16:02:45.548091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:45.548062    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:02:46.104629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:46.104587    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g4jw4" event={"ID":"85669af1-ab91-42af-a131-b3d1c858e737","Type":"ContainerStarted","Data":"d435954aade6506403483d1e7da106782b1eeaf7d2bbdc7e93acff290a1c799f"}
Oct 31 16:02:47.112393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:47.112353    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwx8c" event={"ID":"b04f7600-352a-47ca-8901-98ab6be3dfb2","Type":"ContainerStarted","Data":"82df7f1fe629839babcc983a8b06a6fbb3f6198f07ea6e605d5b0f1b296970bc"}
Oct 31 16:02:47.112835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:47.112415    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xwx8c" event={"ID":"b04f7600-352a-47ca-8901-98ab6be3dfb2","Type":"ContainerStarted","Data":"fa67e4bf62ddd7fd0207ee35cd48f44fc3dd7841397fb0aa9b621cb33186554b"}
Oct 31 16:02:47.112835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:47.112471    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:48.117551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:48.117510    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g4jw4" event={"ID":"85669af1-ab91-42af-a131-b3d1c858e737","Type":"ContainerStarted","Data":"7ad5abc556d5b97399f8bfc30c505d96f185f0bf17ffa04368346093734894fa"}
Oct 31 16:02:48.361973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:48.361942    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n76gt"
Oct 31 16:02:49.894592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:49.894557    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-676bf8f5f9-8ksmg"]
Oct 31 16:02:49.896595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:49.896577    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:49.991247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:49.991220    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 16:02:50.008298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008277    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-bound-sa-token\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008308    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-certificates\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008323    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-trusted-ca\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008353    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-image-registry-private-configuration\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008400    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-installation-pull-secrets\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008455    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96a7bc17-b340-4653-819e-ac1a91c826bb-ca-trust-extracted\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008513    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6d2\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-kube-api-access-dc6d2\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.008578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.008559    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-tls\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.037796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.037767    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 16:02:50.038514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.038492    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676bf8f5f9-8ksmg"]
Oct 31 16:02:50.053624 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.053598    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 16:02:50.053707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.053609    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zwdm2\""
Oct 31 16:02:50.053761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.053715    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 16:02:50.109827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109803    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-certificates\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.109925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109829    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-trusted-ca\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.109925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109866    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-image-registry-private-configuration\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.109925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109895    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-installation-pull-secrets\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.109925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109912    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96a7bc17-b340-4653-819e-ac1a91c826bb-ca-trust-extracted\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.110113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109933    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6d2\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-kube-api-access-dc6d2\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.110113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109961    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-tls\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.110113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.109994    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-bound-sa-token\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.110536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.110515    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96a7bc17-b340-4653-819e-ac1a91c826bb-ca-trust-extracted\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.111025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.110996    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-certificates\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.111125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.111102    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96a7bc17-b340-4653-819e-ac1a91c826bb-trusted-ca\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.112270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.112247    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-installation-pull-secrets\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.112346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.112269    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96a7bc17-b340-4653-819e-ac1a91c826bb-image-registry-private-configuration\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.112394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.112377    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-registry-tls\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.162469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.162417    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-bound-sa-token\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.162644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.162607    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6d2\" (UniqueName: \"kubernetes.io/projected/96a7bc17-b340-4653-819e-ac1a91c826bb-kube-api-access-dc6d2\") pod \"image-registry-676bf8f5f9-8ksmg\" (UID: \"96a7bc17-b340-4653-819e-ac1a91c826bb\") " pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.206455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.206428    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:50.660412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:50.660379    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-676bf8f5f9-8ksmg"]
Oct 31 16:02:50.698670 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:50.698646    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a7bc17_b340_4653_819e_ac1a91c826bb.slice/crio-d9e56a853fd697d6568a122da92a963e5780e1029c5eab65c7e4dc35539a3eff WatchSource:0}: Error finding container d9e56a853fd697d6568a122da92a963e5780e1029c5eab65c7e4dc35539a3eff: Status 404 returned error can't find the container with id d9e56a853fd697d6568a122da92a963e5780e1029c5eab65c7e4dc35539a3eff
Oct 31 16:02:51.127263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.127224    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg" event={"ID":"96a7bc17-b340-4653-819e-ac1a91c826bb","Type":"ContainerStarted","Data":"3b34ccea3adc727b7bffdee9348a68bf3b2678a46c4ad1b9de2e167cfacf66f9"}
Oct 31 16:02:51.127668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.127281    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg" event={"ID":"96a7bc17-b340-4653-819e-ac1a91c826bb","Type":"ContainerStarted","Data":"d9e56a853fd697d6568a122da92a963e5780e1029c5eab65c7e4dc35539a3eff"}
Oct 31 16:02:51.127668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.127322    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:02:51.235653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.235587    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg" podStartSLOduration=2.235574484 podStartE2EDuration="2.235574484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:51.234952398 +0000 UTC m=+16.296527583" watchObservedRunningTime="2026-10-31 16:02:51.235574484 +0000 UTC m=+16.297149669"
Oct 31 16:02:51.923291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.923261    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:51.925477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:51.925455    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4-metrics-certs\") pod \"network-metrics-daemon-7q6q2\" (UID: \"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4\") " pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:52.037075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:52.037042    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7q6q2"
Oct 31 16:02:52.392581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:52.392551    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7q6q2"]
Oct 31 16:02:52.395676 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:52.395651    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a5d76a_bdd6_483e_9ba5_fbcf2c9390b4.slice/crio-4d591cc4758f2037f3dde979c33a2f73f91b621d28c6e842c1671b0e0be48412 WatchSource:0}: Error finding container 4d591cc4758f2037f3dde979c33a2f73f91b621d28c6e842c1671b0e0be48412: Status 404 returned error can't find the container with id 4d591cc4758f2037f3dde979c33a2f73f91b621d28c6e842c1671b0e0be48412
Oct 31 16:02:53.134385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:53.134353    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7q6q2" event={"ID":"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4","Type":"ContainerStarted","Data":"4d591cc4758f2037f3dde979c33a2f73f91b621d28c6e842c1671b0e0be48412"}
Oct 31 16:02:55.141145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.141103    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7q6q2" event={"ID":"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4","Type":"ContainerStarted","Data":"2b280e58790bf45db66dd84b3c180b9d77a3f045e0ed1bd93b3676a0378f3b2e"}
Oct 31 16:02:55.141581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.141194    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7q6q2" event={"ID":"24a5d76a-bdd6-483e-9ba5-fbcf2c9390b4","Type":"ContainerStarted","Data":"9d8cb2c35b58985fe959508b2cfaf9ae04f60b25079065699ec00de75c1c10e9"}
Oct 31 16:02:55.439033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.438956    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"]
Oct 31 16:02:55.442045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.442022    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:55.494997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.494792    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 16:02:55.495264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.495245    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nbdhn\""
Oct 31 16:02:55.495446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.495428    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"]
Oct 31 16:02:55.547913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.547883    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a42dd32c-24e9-4e10-841e-ff0055744450-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nfltz\" (UID: \"a42dd32c-24e9-4e10-841e-ff0055744450\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:55.648665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.648640    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a42dd32c-24e9-4e10-841e-ff0055744450-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nfltz\" (UID: \"a42dd32c-24e9-4e10-841e-ff0055744450\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:55.650918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.650898    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a42dd32c-24e9-4e10-841e-ff0055744450-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nfltz\" (UID: \"a42dd32c-24e9-4e10-841e-ff0055744450\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:55.752694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:55.752639    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:56.107924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:56.107893    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"]
Oct 31 16:02:56.111235 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:02:56.111209    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42dd32c_24e9_4e10_841e_ff0055744450.slice/crio-03f7318e92ea8c7486ad6b131d052af7a383a49aa56ab860b8a68724fcf15374 WatchSource:0}: Error finding container 03f7318e92ea8c7486ad6b131d052af7a383a49aa56ab860b8a68724fcf15374: Status 404 returned error can't find the container with id 03f7318e92ea8c7486ad6b131d052af7a383a49aa56ab860b8a68724fcf15374
Oct 31 16:02:56.145403 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:56.145370    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz" event={"ID":"a42dd32c-24e9-4e10-841e-ff0055744450","Type":"ContainerStarted","Data":"03f7318e92ea8c7486ad6b131d052af7a383a49aa56ab860b8a68724fcf15374"}
Oct 31 16:02:57.120113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:57.120085    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xwx8c"
Oct 31 16:02:58.153007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:58.152973    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz" event={"ID":"a42dd32c-24e9-4e10-841e-ff0055744450","Type":"ContainerStarted","Data":"c254617be3673533a3dbd6e796825df80d00652a05452b158343c921717b6b99"}
Oct 31 16:02:58.153487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:58.153185    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:58.157769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:58.157748    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz"
Oct 31 16:02:58.260630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:02:58.260581    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nfltz" podStartSLOduration=1.615291065 podStartE2EDuration="3.26056857s" totalImagesPullingTime="1.645277505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:58.26015044 +0000 UTC m=+23.321725628" watchObservedRunningTime="2026-10-31 16:02:58.26056857 +0000 UTC m=+23.322143753"
Oct 31 16:03:02.944739 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:03:02.944703    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:03:04.652421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:04.652394    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmvzx"
Oct 31 16:03:05.496995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.496952    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"]
Oct 31 16:03:05.536945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.536921    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8npgm"]
Oct 31 16:03:05.537092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.537045    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.552759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.552732    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5dfz"
Oct 31 16:03:05.552903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.552809    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.553518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.553458    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"]
Oct 31 16:03:05.585237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.585207    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 16:03:05.599011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.598981    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 16:03:05.599114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.599088    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 16:03:05.599228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.599216    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 16:03:05.610346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610323    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610355    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610372    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9js8j\" (UniqueName: \"kubernetes.io/projected/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-api-access-9js8j\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610396    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610475    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610536    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-textfile\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610563    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610578    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-metrics-client-ca\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610597    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-sys\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610632    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610679    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-tls\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610757    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhnj\" (UniqueName: \"kubernetes.io/projected/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-kube-api-access-mmhnj\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610777    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e721f7ad-6663-4c39-bce7-9cbc649ec162-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.610837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610803    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-root\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.610837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.610824    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-wtmp\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.631417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631393    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 16:03:05.631417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631416    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 16:03:05.631551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631402    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 16:03:05.631551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631429    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2sntn\""
Oct 31 16:03:05.631551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631451    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 16:03:05.631676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631585    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jlhfg\""
Oct 31 16:03:05.631676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.631636    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 16:03:05.711246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711212    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-sys\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711250    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711276    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-tls\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711327    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-sys\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711447    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhnj\" (UniqueName: \"kubernetes.io/projected/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-kube-api-access-mmhnj\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711485    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e721f7ad-6663-4c39-bce7-9cbc649ec162-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711508    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-root\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711542    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-wtmp\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711582    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711611    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711656    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9js8j\" (UniqueName: \"kubernetes.io/projected/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-api-access-9js8j\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711659    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-root\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711686    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.711734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711716    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711754    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-textfile\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711787    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711806    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-wtmp\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711811    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-metrics-client-ca\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711864    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e721f7ad-6663-4c39-bce7-9cbc649ec162-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.711942    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.712400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.712361    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-metrics-client-ca\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.712595    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.712769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.712595    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.712874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.712768    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-textfile\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.713898 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.713867    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-tls\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.714150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.714117    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.714437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.714418    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.714516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.714462    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.763911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.763853    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhnj\" (UniqueName: \"kubernetes.io/projected/a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6-kube-api-access-mmhnj\") pod \"node-exporter-8npgm\" (UID: \"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6\") " pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.764051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.764033    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9js8j\" (UniqueName: \"kubernetes.io/projected/e721f7ad-6663-4c39-bce7-9cbc649ec162-kube-api-access-9js8j\") pod \"kube-state-metrics-5dc8d7d89b-r5q7s\" (UID: \"e721f7ad-6663-4c39-bce7-9cbc649ec162\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.887416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.887385    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"
Oct 31 16:03:05.939102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:05.939069    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8npgm"
Oct 31 16:03:05.964818 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:05.964776    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c36f19_c8ef_4f4f_b58f_b871bf33f0c6.slice/crio-97c52a3702ce3753c159a5712a6996d6910ab99a6273d37344f5d3134e1732e2 WatchSource:0}: Error finding container 97c52a3702ce3753c159a5712a6996d6910ab99a6273d37344f5d3134e1732e2: Status 404 returned error can't find the container with id 97c52a3702ce3753c159a5712a6996d6910ab99a6273d37344f5d3134e1732e2
Oct 31 16:03:06.178717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:06.178684    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8npgm" event={"ID":"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6","Type":"ContainerStarted","Data":"97c52a3702ce3753c159a5712a6996d6910ab99a6273d37344f5d3134e1732e2"}
Oct 31 16:03:06.324327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:06.324293    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s"]
Oct 31 16:03:06.327750 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:06.327721    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode721f7ad_6663_4c39_bce7_9cbc649ec162.slice/crio-261495eabfd838f56922cca9280f7b57c69a7d75e68c24252c2efd644911d055 WatchSource:0}: Error finding container 261495eabfd838f56922cca9280f7b57c69a7d75e68c24252c2efd644911d055: Status 404 returned error can't find the container with id 261495eabfd838f56922cca9280f7b57c69a7d75e68c24252c2efd644911d055
Oct 31 16:03:07.183880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.183844    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s" event={"ID":"e721f7ad-6663-4c39-bce7-9cbc649ec162","Type":"ContainerStarted","Data":"261495eabfd838f56922cca9280f7b57c69a7d75e68c24252c2efd644911d055"}
Oct 31 16:03:07.729988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.729951    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:03:07.744315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.744289    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.776291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.776254    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:03:07.792550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.792526    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 16:03:07.792550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.792543    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 16:03:07.792809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.792662    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 16:03:07.792809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.792687    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 16:03:07.793215 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.793193    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 16:03:07.793215 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.793206    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9jhjp\""
Oct 31 16:03:07.793375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.793247    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 16:03:07.793499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.793479    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 16:03:07.826869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.826845    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.826977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.826872    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.826977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.826898    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.826977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.826954    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827043    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llkc\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827066    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827085    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827105    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827169    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827241    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827262    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827279    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.827309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.827298    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.838735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.838719    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 16:03:07.886495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.886467    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 16:03:07.927954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.927921    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.927954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.927956    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.927983    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928098    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928146    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928174    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928203    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928228    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928768    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.928849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928813    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928863    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928897    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llkc\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.928967    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.929008    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.929264    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.929894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.929842    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.931109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.931084    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.931296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.931271    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.931943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.931760    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.931943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.931901    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.932102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.932027    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.932102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.932069    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.932182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.932108    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.932484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.932465    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.933120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.933103    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:07.984779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:07.984717    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llkc\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\") pod \"alertmanager-main-0\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:08.156840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:08.156816    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:03:08.652447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:08.652210    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:03:08.816330 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:08.816291    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cba51ae_42f0_4e4a_bc54_051441edb217.slice/crio-20ac31703a2e9f2bd1c0ff60dae59fe22ee89744cf22d2024edfdd4e9b1a77dd WatchSource:0}: Error finding container 20ac31703a2e9f2bd1c0ff60dae59fe22ee89744cf22d2024edfdd4e9b1a77dd: Status 404 returned error can't find the container with id 20ac31703a2e9f2bd1c0ff60dae59fe22ee89744cf22d2024edfdd4e9b1a77dd
Oct 31 16:03:09.191798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.191720    4417 generic.go:356] "Generic (PLEG): container finished" podID="a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6" containerID="0ddca6f51f97035c925c57da7da83d6d1e88a2f8f3c4e444670b221c1888a205" exitCode=0
Oct 31 16:03:09.191939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.191810    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8npgm" event={"ID":"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6","Type":"ContainerDied","Data":"0ddca6f51f97035c925c57da7da83d6d1e88a2f8f3c4e444670b221c1888a205"}
Oct 31 16:03:09.193082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.193061    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"20ac31703a2e9f2bd1c0ff60dae59fe22ee89744cf22d2024edfdd4e9b1a77dd"}
Oct 31 16:03:09.195126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.195045    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s" event={"ID":"e721f7ad-6663-4c39-bce7-9cbc649ec162","Type":"ContainerStarted","Data":"7b8a2a5d8d9d8958323c9fe5e8eba9bd91c0eb7a5ee5966171e0a1d116841e2c"}
Oct 31 16:03:09.195126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.195088    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s" event={"ID":"e721f7ad-6663-4c39-bce7-9cbc649ec162","Type":"ContainerStarted","Data":"d7faf4472909787913fce362566126dc67a968f0f35cba328044b02e837b73f5"}
Oct 31 16:03:09.195126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:09.195107    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s" event={"ID":"e721f7ad-6663-4c39-bce7-9cbc649ec162","Type":"ContainerStarted","Data":"6f5be0dd3a6807a9e1ac346d6e37cbfad84366665f93835c602f6a9a64a32006"}
Oct 31 16:03:10.201846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:10.201802    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8npgm" event={"ID":"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6","Type":"ContainerStarted","Data":"f33e0f036f501ca366b7f6377f88ff93771f1ac98c71b00687f89569ccbce1a1"}
Oct 31 16:03:10.202256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:10.201872    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8npgm" event={"ID":"a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6","Type":"ContainerStarted","Data":"77e839a468a3f776f06daf8da1f3e78d6a5a897f68a61fff757e15c21f7e38bf"}
Oct 31 16:03:10.322435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:10.322382    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r5q7s" podStartSLOduration=2.804407135 podStartE2EDuration="5.322363837s" totalImagesPullingTime="2.517956702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:09.408054577 +0000 UTC m=+34.469629763" watchObservedRunningTime="2026-10-31 16:03:10.322363837 +0000 UTC m=+35.383939024"
Oct 31 16:03:10.322965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:10.322925    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8npgm" podStartSLOduration=3.104081839 podStartE2EDuration="5.322914514s" totalImagesPullingTime="2.13350251s" totalInitContainerRuntime="85.330165ms" isStatefulPod=true podCreationTimestamp="2026-10-31 16:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:10.32182747 +0000 UTC m=+35.383402680" watchObservedRunningTime="2026-10-31 16:03:10.322914514 +0000 UTC m=+35.384489712"
Oct 31 16:03:11.206476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:11.206444    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9" exitCode=0
Oct 31 16:03:11.206981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:11.206532    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9"}
Oct 31 16:03:12.134527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:12.134500    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-676bf8f5f9-8ksmg"
Oct 31 16:03:14.222357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:14.222327    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"}
Oct 31 16:03:14.222733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:14.222391    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"}
Oct 31 16:03:14.222733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:14.222425    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"}
Oct 31 16:03:14.222733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:14.222450    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"}
Oct 31 16:03:14.222733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:14.222472    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"}
Oct 31 16:03:15.228648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:15.228556    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerStarted","Data":"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"}
Oct 31 16:03:15.482843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:15.482746    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.396981598 podStartE2EDuration="8.482730614s" totalImagesPullingTime="5.074668161s" totalInitContainerRuntime="11.080855ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:15.435767285 +0000 UTC m=+40.497342471" watchObservedRunningTime="2026-10-31 16:03:15.482730614 +0000 UTC m=+40.544305831"
Oct 31 16:03:20.590940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.590864    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:03:20.594775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.594755    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.643528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.643504    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 16:03:20.643648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.643557    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 16:03:20.690086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.690055    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 16:03:20.690086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.690072    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 16:03:20.690290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.690274    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 16:03:20.691131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.691114    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:03:20.692594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.692579    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 16:03:20.729003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.728972    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729007    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729038    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729146    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729201    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729237    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pppll\" (UniqueName: \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.729307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.729293    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.743708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.743673    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 16:03:20.743832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.743793    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cj9z5\""
Oct 31 16:03:20.829714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829675    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829718    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829739    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pppll\" (UniqueName: \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829767    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829798    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829828    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.829954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.829859    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.830287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.830261    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.830557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.830535    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.830664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.830649    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.830719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.830675    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.832410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.832385    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.832531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.832411    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.882542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.882469    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pppll\" (UniqueName: \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\") pod \"console-67f8c4b9d8-qjd28\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") " pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:20.905463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:20.905410    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:21.266283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:21.266202    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:03:21.269163 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:21.269134    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc936fe1d_0d7c_4d02_a876_7a01155850c2.slice/crio-9b1643f5459ad2d48c8db8e43b78af8a427310680c3249ac0687ba8f93f80221 WatchSource:0}: Error finding container 9b1643f5459ad2d48c8db8e43b78af8a427310680c3249ac0687ba8f93f80221: Status 404 returned error can't find the container with id 9b1643f5459ad2d48c8db8e43b78af8a427310680c3249ac0687ba8f93f80221
Oct 31 16:03:22.253230 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:22.253194    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8c4b9d8-qjd28" event={"ID":"c936fe1d-0d7c-4d02-a876-7a01155850c2","Type":"ContainerStarted","Data":"9b1643f5459ad2d48c8db8e43b78af8a427310680c3249ac0687ba8f93f80221"}
Oct 31 16:03:25.265276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:25.265189    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8c4b9d8-qjd28" event={"ID":"c936fe1d-0d7c-4d02-a876-7a01155850c2","Type":"ContainerStarted","Data":"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"}
Oct 31 16:03:25.372061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:25.372008    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-67f8c4b9d8-qjd28" podStartSLOduration=1.721037694 podStartE2EDuration="5.371995121s" totalImagesPullingTime="3.650957427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:25.370943346 +0000 UTC m=+50.432518551" watchObservedRunningTime="2026-10-31 16:03:25.371995121 +0000 UTC m=+50.433570307"
Oct 31 16:03:30.906062 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:30.906021    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:30.906581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:30.906098    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:30.910983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:30.910961    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:31.288844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:31.288767    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:03:32.943221 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:03:32.943184    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:03:34.101822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.101782    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:03:34.107782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.107760    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234156    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234204    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234219    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgck\" (UniqueName: \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234347    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234381    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234405    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234460    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.234515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.234491    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.301299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.297681    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 16:03:34.301299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.301242    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:03:34.334919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.334886    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.334931    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.334952    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgck\" (UniqueName: \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.334974    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.334992    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335013    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335032    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335049    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335734    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335803    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335913    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335916    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.335985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.335974    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.337515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.337494    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.337790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.337765    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.388297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.388224    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgck\" (UniqueName: \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\") pod \"console-5485f6856b-h5vpm\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") " pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.419203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.419173    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:34.781819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:34.781733    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:03:34.784713 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:34.784680    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39d2689_c8fb_43bc_b3a9_4b77140396b2.slice/crio-7d8166c08a37349d5608502b038211e4d1c8ad30310483f7a05ebfd79d80010f WatchSource:0}: Error finding container 7d8166c08a37349d5608502b038211e4d1c8ad30310483f7a05ebfd79d80010f: Status 404 returned error can't find the container with id 7d8166c08a37349d5608502b038211e4d1c8ad30310483f7a05ebfd79d80010f
Oct 31 16:03:35.298571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:35.298535    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5485f6856b-h5vpm" event={"ID":"c39d2689-c8fb-43bc-b3a9-4b77140396b2","Type":"ContainerStarted","Data":"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"}
Oct 31 16:03:35.299063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:35.298592    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5485f6856b-h5vpm" event={"ID":"c39d2689-c8fb-43bc-b3a9-4b77140396b2","Type":"ContainerStarted","Data":"7d8166c08a37349d5608502b038211e4d1c8ad30310483f7a05ebfd79d80010f"}
Oct 31 16:03:35.407421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:35.407362    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5485f6856b-h5vpm" podStartSLOduration=1.407346583 podStartE2EDuration="1.407346583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:35.407263138 +0000 UTC m=+60.468838323" watchObservedRunningTime="2026-10-31 16:03:35.407346583 +0000 UTC m=+60.468921769"
Oct 31 16:03:44.420024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:44.419981    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:44.420519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:44.420059    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:44.424866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:44.424844    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:45.334923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:45.334891    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:03:45.489809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:45.489765    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:03:52.361348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.361311    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f76f78485-979j6"]
Oct 31 16:03:52.367163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.367142    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.458713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.458683    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f76f78485-979j6"]
Oct 31 16:03:52.476851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.476820    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-oauth-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.476851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.476850    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-trusted-ca-bundle\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.476873    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-oauth-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.476891    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.477058    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cvvp\" (UniqueName: \"kubernetes.io/projected/14daeac2-864c-401d-882f-2e7e241c874c-kube-api-access-8cvvp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.477126    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-service-ca\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.477200    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-console-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.477266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.477231    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14daeac2-864c-401d-882f-2e7e241c874c-tmp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.577907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.577875    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cvvp\" (UniqueName: \"kubernetes.io/projected/14daeac2-864c-401d-882f-2e7e241c874c-kube-api-access-8cvvp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.577917    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-service-ca\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.577947    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-console-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.577962    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14daeac2-864c-401d-882f-2e7e241c874c-tmp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.577999    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-oauth-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578016    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-trusted-ca-bundle\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578037    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-oauth-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578059    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578430    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14daeac2-864c-401d-882f-2e7e241c874c-tmp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578707    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-service-ca\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578818    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-oauth-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578917    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-console-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.578972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.578924    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14daeac2-864c-401d-882f-2e7e241c874c-trusted-ca-bundle\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.580505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.580486    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-oauth-config\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.580678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.580662    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14daeac2-864c-401d-882f-2e7e241c874c-console-serving-cert\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.632241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.632175    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cvvp\" (UniqueName: \"kubernetes.io/projected/14daeac2-864c-401d-882f-2e7e241c874c-kube-api-access-8cvvp\") pod \"console-5f76f78485-979j6\" (UID: \"14daeac2-864c-401d-882f-2e7e241c874c\") " pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:52.681656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:52.681611    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:03:53.046494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:53.046465    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f76f78485-979j6"]
Oct 31 16:03:53.047190 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:03:53.047165    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14daeac2_864c_401d_882f_2e7e241c874c.slice/crio-ce389e3b71f852166fdb4192ec31501b0ab6c9c0fc04636a1a8cdf8c9b945df3 WatchSource:0}: Error finding container ce389e3b71f852166fdb4192ec31501b0ab6c9c0fc04636a1a8cdf8c9b945df3: Status 404 returned error can't find the container with id ce389e3b71f852166fdb4192ec31501b0ab6c9c0fc04636a1a8cdf8c9b945df3
Oct 31 16:03:53.357986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:53.357905    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76f78485-979j6" event={"ID":"14daeac2-864c-401d-882f-2e7e241c874c","Type":"ContainerStarted","Data":"774450317bb675aa82e9b3d066c78e7cd5f2254881cb2713ced8b8c8f876f51c"}
Oct 31 16:03:53.357986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:53.357952    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f76f78485-979j6" event={"ID":"14daeac2-864c-401d-882f-2e7e241c874c","Type":"ContainerStarted","Data":"ce389e3b71f852166fdb4192ec31501b0ab6c9c0fc04636a1a8cdf8c9b945df3"}
Oct 31 16:03:53.467772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:53.467713    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f76f78485-979j6" podStartSLOduration=1.467698708 podStartE2EDuration="1.467698708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:53.466553718 +0000 UTC m=+78.528128904" watchObservedRunningTime="2026-10-31 16:03:53.467698708 +0000 UTC m=+78.529273929"
Oct 31 16:03:59.319632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.319581    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:03:59.320050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.319903    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="alertmanager" containerID="cri-o://d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02" gracePeriod=120
Oct 31 16:03:59.320050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.319966    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-metric" containerID="cri-o://7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243" gracePeriod=120
Oct 31 16:03:59.320050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.319985    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-web" containerID="cri-o://6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b" gracePeriod=120
Oct 31 16:03:59.320222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.320020    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="config-reloader" containerID="cri-o://5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31" gracePeriod=120
Oct 31 16:03:59.320222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.320033    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy" containerID="cri-o://fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411" gracePeriod=120
Oct 31 16:03:59.320222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:03:59.320087    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="prom-label-proxy" containerID="cri-o://d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4" gracePeriod=120
Oct 31 16:04:00.388574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388546    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4" exitCode=0
Oct 31 16:04:00.388574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388571    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243" exitCode=0
Oct 31 16:04:00.388574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388579    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411" exitCode=0
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388585    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31" exitCode=0
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388591    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02" exitCode=0
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388635    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"}
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388755    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"}
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388800    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"}
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388840    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"}
Oct 31 16:04:00.389054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.388881    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"}
Oct 31 16:04:00.585933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.585911    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:00.635961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.635896    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.635961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.635938    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.635963    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636000    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636020    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636053    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636092    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636123    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636156    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636201    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636235    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636258    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636298    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\") pod \"9cba51ae-42f0-4e4a-bc54-051441edb217\" (UID: \"9cba51ae-42f0-4e4a-bc54-051441edb217\") "
Oct 31 16:04:00.636365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636347    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:00.636751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636575    4417 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-main-db\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.637251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.636818    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:00.637529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.637504    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:00.639115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.639085    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.639374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.639347    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.639538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.639512    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.639630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.639594    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.639849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.639828    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.640073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.640056    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:00.640274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.640245    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:00.641187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.641165    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "kube-api-access-7llkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:00.643109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.643087    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.649845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.649823    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config" pod "9cba51ae-42f0-4e4a-bc54-051441edb217" (UID: "9cba51ae-42f0-4e4a-bc54-051441edb217"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:00.737745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737724    4417 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-web-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737744    4417 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9cba51ae-42f0-4e4a-bc54-051441edb217-config-out\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737756    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llkc\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-kube-api-access-7llkc\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737766    4417 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737775    4417 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737784    4417 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737792    4417 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-cluster-tls-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737804    4417 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-main-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737817    4417 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9cba51ae-42f0-4e4a-bc54-051441edb217-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737827    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-metrics-client-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737835    4417 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cba51ae-42f0-4e4a-bc54-051441edb217-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:00.737888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:00.737843    4417 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9cba51ae-42f0-4e4a-bc54-051441edb217-tls-assets\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:01.395141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.395111    4417 generic.go:356] "Generic (PLEG): container finished" podID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerID="6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b" exitCode=0
Oct 31 16:04:01.395585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.395168    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"}
Oct 31 16:04:01.395585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.395197    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.395585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.395245    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9cba51ae-42f0-4e4a-bc54-051441edb217","Type":"ContainerDied","Data":"20ac31703a2e9f2bd1c0ff60dae59fe22ee89744cf22d2024edfdd4e9b1a77dd"}
Oct 31 16:04:01.395585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.395299    4417 scope.go:122] "RemoveContainer" containerID="d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"
Oct 31 16:04:01.402637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.402603    4417 scope.go:122] "RemoveContainer" containerID="7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"
Oct 31 16:04:01.409005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.408986    4417 scope.go:122] "RemoveContainer" containerID="fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"
Oct 31 16:04:01.415059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.415042    4417 scope.go:122] "RemoveContainer" containerID="6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"
Oct 31 16:04:01.421350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.421333    4417 scope.go:122] "RemoveContainer" containerID="5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"
Oct 31 16:04:01.427549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.427529    4417 scope.go:122] "RemoveContainer" containerID="d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"
Oct 31 16:04:01.433921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.433903    4417 scope.go:122] "RemoveContainer" containerID="8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9"
Oct 31 16:04:01.440238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440093    4417 scope.go:122] "RemoveContainer" containerID="d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"
Oct 31 16:04:01.440355 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.440335    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4\": container with ID starting with d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4 not found: ID does not exist" containerID="d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"
Oct 31 16:04:01.440397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440364    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4"} err="failed to get container status \"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4\": rpc error: code = NotFound desc = could not find container \"d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4\": container with ID starting with d9642ea98d062eab9491f48ce7c2e6c02b5b24565801ae9b56806a20c8b268f4 not found: ID does not exist"
Oct 31 16:04:01.440437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440398    4417 scope.go:122] "RemoveContainer" containerID="7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"
Oct 31 16:04:01.440658 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.440638    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243\": container with ID starting with 7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243 not found: ID does not exist" containerID="7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"
Oct 31 16:04:01.440707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440667    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243"} err="failed to get container status \"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243\": rpc error: code = NotFound desc = could not find container \"7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243\": container with ID starting with 7cf4b376a12233c2d139ae7507771b9d8b64a2669c7ca5e33ff57c4a213a5243 not found: ID does not exist"
Oct 31 16:04:01.440707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440683    4417 scope.go:122] "RemoveContainer" containerID="fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"
Oct 31 16:04:01.440895 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.440873    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411\": container with ID starting with fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411 not found: ID does not exist" containerID="fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"
Oct 31 16:04:01.440940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440892    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411"} err="failed to get container status \"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411\": rpc error: code = NotFound desc = could not find container \"fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411\": container with ID starting with fe54f6fb6b411bf3af3e3846a30ae10b4f7397f620783ca58de6b85a1cdb3411 not found: ID does not exist"
Oct 31 16:04:01.440940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.440904    4417 scope.go:122] "RemoveContainer" containerID="6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"
Oct 31 16:04:01.441109 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.441095    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b\": container with ID starting with 6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b not found: ID does not exist" containerID="6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"
Oct 31 16:04:01.441143 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441112    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b"} err="failed to get container status \"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b\": rpc error: code = NotFound desc = could not find container \"6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b\": container with ID starting with 6e8332af8668ebf1451188c3728388c29631548656ea9d79d4f56dbdf8b4f29b not found: ID does not exist"
Oct 31 16:04:01.441143 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441123    4417 scope.go:122] "RemoveContainer" containerID="5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"
Oct 31 16:04:01.441339 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.441324    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31\": container with ID starting with 5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31 not found: ID does not exist" containerID="5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"
Oct 31 16:04:01.441373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441342    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31"} err="failed to get container status \"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31\": rpc error: code = NotFound desc = could not find container \"5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31\": container with ID starting with 5f33d44b29c11478db45e6671c2dbd03d1e1606378d4617b7cb8377a8e8cca31 not found: ID does not exist"
Oct 31 16:04:01.441373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441354    4417 scope.go:122] "RemoveContainer" containerID="d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"
Oct 31 16:04:01.441564 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.441546    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02\": container with ID starting with d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02 not found: ID does not exist" containerID="d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"
Oct 31 16:04:01.441603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441569    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02"} err="failed to get container status \"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02\": rpc error: code = NotFound desc = could not find container \"d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02\": container with ID starting with d64af02e780a77103224382879ff77fa0728281f39b9f5f316166c99ec73ed02 not found: ID does not exist"
Oct 31 16:04:01.441603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441581    4417 scope.go:122] "RemoveContainer" containerID="8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9"
Oct 31 16:04:01.441859 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:01.441826    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9\": container with ID starting with 8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9 not found: ID does not exist" containerID="8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9"
Oct 31 16:04:01.441915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.441867    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9"} err="failed to get container status \"8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9\": rpc error: code = NotFound desc = could not find container \"8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9\": container with ID starting with 8f152c4b613f9cbea80b0502cc4051c4175c6c5f3472f7609f19137b8901ebb9 not found: ID does not exist"
Oct 31 16:04:01.596255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.596219    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:04:01.642493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.642415    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:04:01.643377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643356    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:04:01.643597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643586    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="alertmanager"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643598    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="alertmanager"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643608    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643626    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643633    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-metric"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643639    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-metric"
Oct 31 16:04:01.643663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643661    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="init-config-reloader"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643668    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="init-config-reloader"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643675    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="config-reloader"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643680    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="config-reloader"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643685    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-web"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643689    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-web"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643709    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="prom-label-proxy"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643713    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="prom-label-proxy"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643752    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="alertmanager"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643758    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643763    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-metric"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643769    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="config-reloader"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643775    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="kube-rbac-proxy-web"
Oct 31 16:04:01.643837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.643783    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" containerName="prom-label-proxy"
Oct 31 16:04:01.648242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.648226    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.697601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697567    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9jhjp\""
Oct 31 16:04:01.697601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697585    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 16:04:01.697821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697567    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 16:04:01.697821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697649    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 16:04:01.697821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697661    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 16:04:01.697821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697654    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 16:04:01.697821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697686    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 16:04:01.697995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697870    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 16:04:01.697995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.697946    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 16:04:01.743766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.743742    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 16:04:01.744035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744019    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744044    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hsq\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-kube-api-access-x4hsq\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744067    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744093    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744172    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744219    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744291    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-config-out\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744324    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744346    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-web-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744428    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744463    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744490    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.744596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.744512    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845412    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845444    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-web-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845471    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845498    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845535    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845567    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845594    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845639    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hsq\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-kube-api-access-x4hsq\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845670    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.845701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845691    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.846029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845716    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.846029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845740    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.846029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.845787    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-config-out\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.846950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.846918    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.847222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.847198    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.848777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.848752    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.848886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.848770    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ee602f0-74c4-43c8-a904-49ee517ce276-config-out\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.848886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.848865    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-web-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.848886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.848879    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.849055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.848992    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.849224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.849204    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.849293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.849218    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.849363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.849342    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ee602f0-74c4-43c8-a904-49ee517ce276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.850431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.850411    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ee602f0-74c4-43c8-a904-49ee517ce276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.850741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.850725    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.861336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.861314    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:04:01.900259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.900192    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hsq\" (UniqueName: \"kubernetes.io/projected/3ee602f0-74c4-43c8-a904-49ee517ce276-kube-api-access-x4hsq\") pod \"alertmanager-main-0\" (UID: \"3ee602f0-74c4-43c8-a904-49ee517ce276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:01.971359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:01.971328    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 16:04:02.387681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:02.387658    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 16:04:02.389989 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:04:02.389958    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee602f0_74c4_43c8_a904_49ee517ce276.slice/crio-2b0804115ef83a4b2da0585917b0ef19a0b5a0235fa22316e39a73f124a98d17 WatchSource:0}: Error finding container 2b0804115ef83a4b2da0585917b0ef19a0b5a0235fa22316e39a73f124a98d17: Status 404 returned error can't find the container with id 2b0804115ef83a4b2da0585917b0ef19a0b5a0235fa22316e39a73f124a98d17
Oct 31 16:04:02.400315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:02.400285    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"2b0804115ef83a4b2da0585917b0ef19a0b5a0235fa22316e39a73f124a98d17"}
Oct 31 16:04:02.682188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:02.682106    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:04:02.682493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:02.682470    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:04:02.686943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:02.686923    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:04:02.942750 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:02.942672    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:04:03.046841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:03.046809    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cba51ae-42f0-4e4a-bc54-051441edb217" path="/var/lib/kubelet/pods/9cba51ae-42f0-4e4a-bc54-051441edb217/volumes"
Oct 31 16:04:03.405117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:03.405085    4417 generic.go:356] "Generic (PLEG): container finished" podID="3ee602f0-74c4-43c8-a904-49ee517ce276" containerID="332aaa658c3b1da9683369a2f7dda99645c1e2819b72b8441ea1e92541914b97" exitCode=0
Oct 31 16:04:03.405582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:03.405171    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerDied","Data":"332aaa658c3b1da9683369a2f7dda99645c1e2819b72b8441ea1e92541914b97"}
Oct 31 16:04:03.409227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:03.409205    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f76f78485-979j6"
Oct 31 16:04:03.650244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:03.649519    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:04:04.412351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412313    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"73c20f9e9d84e7a1bd3657a438356dcb9030bed5b46f8d30a10fc7e96335787d"}
Oct 31 16:04:04.412758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412399    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"336d54d0c25bfca132b34736f889705a9a42fb35c61ac8d7a63bf09bf8ac3c2b"}
Oct 31 16:04:04.412758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412441    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"53740cef12be6c2e27430a6db9d7e9ecd73812834e9236967c5b8d8a61d60cb9"}
Oct 31 16:04:04.412758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412481    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"e0d3f5f38c80217867b91b81b813ada09d7c1104b5a380b721ad10c74d83dfd0"}
Oct 31 16:04:04.412758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412519    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"a012fff6338810416e29a959a3257d7e52e122d7f713f7a811e34bca4e869ac1"}
Oct 31 16:04:04.412758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.412554    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ee602f0-74c4-43c8-a904-49ee517ce276","Type":"ContainerStarted","Data":"5936a0dd051c547c96acca5fa2f7dffebdb3fdaff53961500216adbc7f47c936"}
Oct 31 16:04:04.526075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:04.526032    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.511712191 podStartE2EDuration="3.526018457s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.306266ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:04:04.524692353 +0000 UTC m=+89.586267540" watchObservedRunningTime="2026-10-31 16:04:04.526018457 +0000 UTC m=+89.587593665"
Oct 31 16:04:10.509479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.509406    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-67f8c4b9d8-qjd28" podUID="c936fe1d-0d7c-4d02-a876-7a01155850c2" containerName="console" containerID="cri-o://7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878" gracePeriod=15
Oct 31 16:04:10.754694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.754671    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:04:10.819248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819221    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819258    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819288    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819357    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\" (UniqueName: \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819402    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819556    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819642    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\") pod \"c936fe1d-0d7c-4d02-a876-7a01155850c2\" (UID: \"c936fe1d-0d7c-4d02-a876-7a01155850c2\") "
Oct 31 16:04:10.819752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819704    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:10.819874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819852    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:10.819945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819869    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:10.819945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.819864    4417 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-oauth-serving-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.820123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.820103    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:10.821590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.821564    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:10.821880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.821861    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:10.821950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.821885    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll" pod "c936fe1d-0d7c-4d02-a876-7a01155850c2" (UID: "c936fe1d-0d7c-4d02-a876-7a01155850c2"). InnerVolumeSpecName "kube-api-access-pppll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:10.920658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920598    4417 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.920658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920651    4417 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-serving-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.920658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920661    4417 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c936fe1d-0d7c-4d02-a876-7a01155850c2-console-oauth-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.920883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920670    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pppll\" (UniqueName: \"kubernetes.io/projected/c936fe1d-0d7c-4d02-a876-7a01155850c2-kube-api-access-pppll\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.920883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920682    4417 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c936fe1d-0d7c-4d02-a876-7a01155850c2-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:10.920883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:10.920691    4417 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c936fe1d-0d7c-4d02-a876-7a01155850c2-tmp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:11.443966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.443934    4417 generic.go:356] "Generic (PLEG): container finished" podID="c936fe1d-0d7c-4d02-a876-7a01155850c2" containerID="7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878" exitCode=0
Oct 31 16:04:11.444141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.443975    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8c4b9d8-qjd28" event={"ID":"c936fe1d-0d7c-4d02-a876-7a01155850c2","Type":"ContainerDied","Data":"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"}
Oct 31 16:04:11.444141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.443978    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f8c4b9d8-qjd28"
Oct 31 16:04:11.444141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.444015    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f8c4b9d8-qjd28" event={"ID":"c936fe1d-0d7c-4d02-a876-7a01155850c2","Type":"ContainerDied","Data":"9b1643f5459ad2d48c8db8e43b78af8a427310680c3249ac0687ba8f93f80221"}
Oct 31 16:04:11.444141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.444034    4417 scope.go:122] "RemoveContainer" containerID="7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"
Oct 31 16:04:11.451657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.451547    4417 scope.go:122] "RemoveContainer" containerID="7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"
Oct 31 16:04:11.451862 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:11.451845    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878\": container with ID starting with 7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878 not found: ID does not exist" containerID="7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"
Oct 31 16:04:11.451917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.451870    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878"} err="failed to get container status \"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878\": rpc error: code = NotFound desc = could not find container \"7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878\": container with ID starting with 7ab64f9303438745858f2f8fc676015df89d34096f80e155d5b8b47f6633f878 not found: ID does not exist"
Oct 31 16:04:11.594020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.593985    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:04:11.640552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:11.640498    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f8c4b9d8-qjd28"]
Oct 31 16:04:13.045634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:13.045588    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c936fe1d-0d7c-4d02-a876-7a01155850c2" path="/var/lib/kubelet/pods/c936fe1d-0d7c-4d02-a876-7a01155850c2/volumes"
Oct 31 16:04:28.670051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:28.669988    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5485f6856b-h5vpm" podUID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" containerName="console" containerID="cri-o://bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34" gracePeriod=15
Oct 31 16:04:28.906091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:28.906064    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:04:29.043389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043368    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\" (UniqueName: \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043417    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043445    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043464    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043501    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043538    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043575    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043669    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\") pod \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\" (UID: \"c39d2689-c8fb-43bc-b3a9-4b77140396b2\") "
Oct 31 16:04:29.043962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043929    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:29.043962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.043940    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:29.044250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.044225    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:29.044357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.044230    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:29.044357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.044282    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:29.045743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.045715    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:29.045855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.045774    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "kube-api-access-6qgck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:29.046026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.045999    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config" pod "c39d2689-c8fb-43bc-b3a9-4b77140396b2" (UID: "c39d2689-c8fb-43bc-b3a9-4b77140396b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:29.144345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144313    4417 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c39d2689-c8fb-43bc-b3a9-4b77140396b2-tmp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144340    4417 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-trusted-ca-bundle\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144353    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qgck\" (UniqueName: \"kubernetes.io/projected/c39d2689-c8fb-43bc-b3a9-4b77140396b2-kube-api-access-6qgck\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144366    4417 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144377    4417 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-oauth-serving-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144389    4417 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144400    4417 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-serving-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.144537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.144412    4417 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c39d2689-c8fb-43bc-b3a9-4b77140396b2-console-oauth-config\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:04:29.506286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.506202    4417 generic.go:356] "Generic (PLEG): container finished" podID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" containerID="bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34" exitCode=0
Oct 31 16:04:29.506286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.506246    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5485f6856b-h5vpm"
Oct 31 16:04:29.506286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.506275    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5485f6856b-h5vpm" event={"ID":"c39d2689-c8fb-43bc-b3a9-4b77140396b2","Type":"ContainerDied","Data":"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"}
Oct 31 16:04:29.506555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.506324    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5485f6856b-h5vpm" event={"ID":"c39d2689-c8fb-43bc-b3a9-4b77140396b2","Type":"ContainerDied","Data":"7d8166c08a37349d5608502b038211e4d1c8ad30310483f7a05ebfd79d80010f"}
Oct 31 16:04:29.506555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.506349    4417 scope.go:122] "RemoveContainer" containerID="bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"
Oct 31 16:04:29.513853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.513825    4417 scope.go:122] "RemoveContainer" containerID="bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"
Oct 31 16:04:29.514122 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:29.514106    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34\": container with ID starting with bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34 not found: ID does not exist" containerID="bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"
Oct 31 16:04:29.514168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.514130    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34"} err="failed to get container status \"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34\": rpc error: code = NotFound desc = could not find container \"bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34\": container with ID starting with bc567a05596b9ae6b1811f8ea286e60fbedc0aca07e28645e7de93b5be729a34 not found: ID does not exist"
Oct 31 16:04:29.657028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.657003    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:04:29.703092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:29.703072    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5485f6856b-h5vpm"]
Oct 31 16:04:31.046511 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:04:31.046476    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" path="/var/lib/kubelet/pods/c39d2689-c8fb-43bc-b3a9-4b77140396b2/volumes"
Oct 31 16:04:32.956196 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:04:32.956157    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:05:02.942419 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:05:02.942341    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:05:32.943367 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:05:32.943329    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:05:51.430265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430235    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430490    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c936fe1d-0d7c-4d02-a876-7a01155850c2" containerName="console"
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430500    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c936fe1d-0d7c-4d02-a876-7a01155850c2" containerName="console"
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430524    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" containerName="console"
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430528    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" containerName="console"
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430565    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39d2689-c8fb-43bc-b3a9-4b77140396b2" containerName="console"
Oct 31 16:05:51.430732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.430573    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c936fe1d-0d7c-4d02-a876-7a01155850c2" containerName="console"
Oct 31 16:05:51.433318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.433301    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.482074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.482039    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 16:05:51.482196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.482104    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 16:05:51.529513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.529487    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:05:51.544100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544068    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544107    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544127    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544247    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544263    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544333 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544290    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq89c\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.544366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.544346    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.558882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.558857    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 16:05:51.558987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.558858    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 16:05:51.558987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.558906    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 16:05:51.558987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.558927    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2bg6m\""
Oct 31 16:05:51.582786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.582764    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 16:05:51.645378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645351    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq89c\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645383    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645427    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645448    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645467    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645489    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.645778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.645503    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.646167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.646141    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.647890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.647863    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.647969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.647897    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.648067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.648047    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.648106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.648063    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.700778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.700727    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.700957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.700929    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq89c\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\") pod \"istiod-openshift-gateway-7876bfcb74-mhsg7\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:51.744314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:51.744296    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:52.203042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:52.203011    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:05:52.206495 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:05:52.206460    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01370d6_1248_43b5_b53e_b28d32750cd0.slice/crio-27c82b5e395a187d4fe170fe4cb60b7fed2646842500e579ace149c3945caf48 WatchSource:0}: Error finding container 27c82b5e395a187d4fe170fe4cb60b7fed2646842500e579ace149c3945caf48: Status 404 returned error can't find the container with id 27c82b5e395a187d4fe170fe4cb60b7fed2646842500e579ace149c3945caf48
Oct 31 16:05:52.783715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:52.783680    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" event={"ID":"e01370d6-1248-43b5-b53e-b28d32750cd0","Type":"ContainerStarted","Data":"27c82b5e395a187d4fe170fe4cb60b7fed2646842500e579ace149c3945caf48"}
Oct 31 16:05:56.241672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:56.241635    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:05:56.241962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:56.241708    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:05:56.799606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:56.799577    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" event={"ID":"e01370d6-1248-43b5-b53e-b28d32750cd0","Type":"ContainerStarted","Data":"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"}
Oct 31 16:05:56.799854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:56.799812    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:05:56.955909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:56.955611    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" podStartSLOduration=1.923222129 podStartE2EDuration="5.955593347s" totalImagesPullingTime="4.032371218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:56.954940999 +0000 UTC m=+202.016516184" watchObservedRunningTime="2026-10-31 16:05:56.955593347 +0000 UTC m=+202.017168534"
Oct 31 16:05:57.805374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:57.805339    4417 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mhsg7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:05:57.805791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:05:57.805400    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:06:00.804667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:00.804634    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:06:02.943661 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:06:02.943609    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:06:32.942893 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:06:32.942859    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:06:51.046451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.046421    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"]
Oct 31 16:06:51.049677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.049661    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.070744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070720    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvbv\" (UniqueName: \"kubernetes.io/projected/defad208-e885-463a-8a3f-b253e7a06d32-kube-api-access-wmvbv\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.070869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070771    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.070869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070825    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.070869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070846    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.071016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070875    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.071016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.070936    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.165696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.165665    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"]
Oct 31 16:06:51.172149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172120    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172254    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172305    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvbv\" (UniqueName: \"kubernetes.io/projected/defad208-e885-463a-8a3f-b253e7a06d32-kube-api-access-wmvbv\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172383    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172417    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172451    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.172820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.172798    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.174662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.174640    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/defad208-e885-463a-8a3f-b253e7a06d32-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.187445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.187425    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 16:06:51.187960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.187944    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zc6f5\""
Oct 31 16:06:51.188302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.188288    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 16:06:51.188501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.188489    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 16:06:51.188837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.188824    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 16:06:51.189094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.189081    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 16:06:51.195043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.194998    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.195145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.195125    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.195204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.195158    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/defad208-e885-463a-8a3f-b253e7a06d32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.226378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.226354    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvbv\" (UniqueName: \"kubernetes.io/projected/defad208-e885-463a-8a3f-b253e7a06d32-kube-api-access-wmvbv\") pod \"opendatahub-operator-controller-manager-54f884c9ff-t257l\" (UID: \"defad208-e885-463a-8a3f-b253e7a06d32\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.364335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.364246    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:51.833665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.833578    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"]
Oct 31 16:06:51.836377 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:06:51.836347    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefad208_e885_463a_8a3f_b253e7a06d32.slice/crio-3c4142bccd04d2ee5fde1f04f0292b78191d2ae4ee1ebf5a7323f1ee99130c94 WatchSource:0}: Error finding container 3c4142bccd04d2ee5fde1f04f0292b78191d2ae4ee1ebf5a7323f1ee99130c94: Status 404 returned error can't find the container with id 3c4142bccd04d2ee5fde1f04f0292b78191d2ae4ee1ebf5a7323f1ee99130c94
Oct 31 16:06:51.987607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:51.987570    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l" event={"ID":"defad208-e885-463a-8a3f-b253e7a06d32","Type":"ContainerStarted","Data":"3c4142bccd04d2ee5fde1f04f0292b78191d2ae4ee1ebf5a7323f1ee99130c94"}
Oct 31 16:06:56.003845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:56.003763    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l" event={"ID":"defad208-e885-463a-8a3f-b253e7a06d32","Type":"ContainerStarted","Data":"41493a519bd122b5912787d865ad018dbca0331fc35bd9d9daf7402cab72caac"}
Oct 31 16:06:57.008947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:57.008909    4417 generic.go:356] "Generic (PLEG): container finished" podID="defad208-e885-463a-8a3f-b253e7a06d32" containerID="41493a519bd122b5912787d865ad018dbca0331fc35bd9d9daf7402cab72caac" exitCode=0
Oct 31 16:06:57.009330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:57.008990    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l" event={"ID":"defad208-e885-463a-8a3f-b253e7a06d32","Type":"ContainerDied","Data":"41493a519bd122b5912787d865ad018dbca0331fc35bd9d9daf7402cab72caac"}
Oct 31 16:06:58.014059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:58.014022    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l" event={"ID":"defad208-e885-463a-8a3f-b253e7a06d32","Type":"ContainerStarted","Data":"b39a54df24d86737b2e5a0434917de608db9ab4e35978b9e9da80ae9285d4f7b"}
Oct 31 16:06:58.014445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:58.014129    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:06:58.228850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:06:58.228677    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l" podStartSLOduration=3.20014132 podStartE2EDuration="8.22865883s" totalImagesPullingTime="4.593340305s" totalInitContainerRuntime="435.177205ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:06:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:58.181738898 +0000 UTC m=+263.243314104" watchObservedRunningTime="2026-10-31 16:06:58.22865883 +0000 UTC m=+263.290234018"
Oct 31 16:07:02.942914 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:07:02.942875    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:07:09.020172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:09.020139    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-t257l"
Oct 31 16:07:25.172049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.172021    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-pfr86"]
Oct 31 16:07:25.175503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.175465    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.271118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.271090    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 16:07:25.271118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.271111    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 16:07:25.271118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.271111    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cc559\""
Oct 31 16:07:25.272263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.272238    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-pfr86"]
Oct 31 16:07:25.361629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.361585    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d30e826-29bf-4bbc-929e-3767701bb10c-tls-certs\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.361756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.361724    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlpk\" (UniqueName: \"kubernetes.io/projected/3d30e826-29bf-4bbc-929e-3767701bb10c-kube-api-access-9tlpk\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.361797 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.361774    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d30e826-29bf-4bbc-929e-3767701bb10c-tmp\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.462473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.462411    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlpk\" (UniqueName: \"kubernetes.io/projected/3d30e826-29bf-4bbc-929e-3767701bb10c-kube-api-access-9tlpk\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.462473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.462449    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d30e826-29bf-4bbc-929e-3767701bb10c-tmp\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.462645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.462490    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d30e826-29bf-4bbc-929e-3767701bb10c-tls-certs\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.464689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.464668    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d30e826-29bf-4bbc-929e-3767701bb10c-tls-certs\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.464900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.464883    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d30e826-29bf-4bbc-929e-3767701bb10c-tmp\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.517766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.517741    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlpk\" (UniqueName: \"kubernetes.io/projected/3d30e826-29bf-4bbc-929e-3767701bb10c-kube-api-access-9tlpk\") pod \"kube-auth-proxy-744f97f697-pfr86\" (UID: \"3d30e826-29bf-4bbc-929e-3767701bb10c\") " pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:25.787675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:25.787579    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86"
Oct 31 16:07:26.149650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:26.149589    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-744f97f697-pfr86"]
Oct 31 16:07:26.153340 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:07:26.153307    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d30e826_29bf_4bbc_929e_3767701bb10c.slice/crio-c872481fb2e2fb871ddae8bf180d32b4bf04a4f9fdb2b37d38053a5ed592960f WatchSource:0}: Error finding container c872481fb2e2fb871ddae8bf180d32b4bf04a4f9fdb2b37d38053a5ed592960f: Status 404 returned error can't find the container with id c872481fb2e2fb871ddae8bf180d32b4bf04a4f9fdb2b37d38053a5ed592960f
Oct 31 16:07:27.114629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:27.114590    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86" event={"ID":"3d30e826-29bf-4bbc-929e-3767701bb10c","Type":"ContainerStarted","Data":"c872481fb2e2fb871ddae8bf180d32b4bf04a4f9fdb2b37d38053a5ed592960f"}
Oct 31 16:07:29.124465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:29.124433    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86" event={"ID":"3d30e826-29bf-4bbc-929e-3767701bb10c","Type":"ContainerStarted","Data":"9679ecebd1b506a8c7d0e0ef5120b2372eed287f4dcc9b2fe276abbbd48fc23d"}
Oct 31 16:07:29.232137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:29.232086    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-744f97f697-pfr86" podStartSLOduration=1.332385939 podStartE2EDuration="4.232073668s" totalImagesPullingTime="2.899687729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:29.231200232 +0000 UTC m=+294.292775419" watchObservedRunningTime="2026-10-31 16:07:29.232073668 +0000 UTC m=+294.293648853"
Oct 31 16:07:32.943519 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:07:32.943484    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:07:33.342744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.342709    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"]
Oct 31 16:07:33.345998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.345978    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.397956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.397925    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"]
Oct 31 16:07:33.423870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.423841    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/138006b7-3c26-4668-b8ab-eb13c70406a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.424029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.423897    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/138006b7-3c26-4668-b8ab-eb13c70406a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.424029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.423995    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsbh\" (UniqueName: \"kubernetes.io/projected/138006b7-3c26-4668-b8ab-eb13c70406a2-kube-api-access-6bsbh\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.442940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.442903    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 16:07:33.442940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.442904    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 16:07:33.443174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.442945    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4r6xh\""
Oct 31 16:07:33.443174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.443008    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 16:07:33.443174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.443039    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 16:07:33.524908 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.524882    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/138006b7-3c26-4668-b8ab-eb13c70406a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.525085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.524919    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/138006b7-3c26-4668-b8ab-eb13c70406a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.525085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.524938    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsbh\" (UniqueName: \"kubernetes.io/projected/138006b7-3c26-4668-b8ab-eb13c70406a2-kube-api-access-6bsbh\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.525681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.525652    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/138006b7-3c26-4668-b8ab-eb13c70406a2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.527667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.527646    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/138006b7-3c26-4668-b8ab-eb13c70406a2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.578135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.578107    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsbh\" (UniqueName: \"kubernetes.io/projected/138006b7-3c26-4668-b8ab-eb13c70406a2-kube-api-access-6bsbh\") pod \"kuadrant-console-plugin-66c4fb575d-c4h8w\" (UID: \"138006b7-3c26-4668-b8ab-eb13c70406a2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:33.670985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:33.670903    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"
Oct 31 16:07:34.029677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:34.029566    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w"]
Oct 31 16:07:34.033254 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:07:34.033226    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod138006b7_3c26_4668_b8ab_eb13c70406a2.slice/crio-e3d6783a4c5c28b748335daae59c953f9fe81a550ee2eedd2d7dbf70f76ab801 WatchSource:0}: Error finding container e3d6783a4c5c28b748335daae59c953f9fe81a550ee2eedd2d7dbf70f76ab801: Status 404 returned error can't find the container with id e3d6783a4c5c28b748335daae59c953f9fe81a550ee2eedd2d7dbf70f76ab801
Oct 31 16:07:34.142564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:34.142530    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w" event={"ID":"138006b7-3c26-4668-b8ab-eb13c70406a2","Type":"ContainerStarted","Data":"e3d6783a4c5c28b748335daae59c953f9fe81a550ee2eedd2d7dbf70f76ab801"}
Oct 31 16:07:35.807907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.807861    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:07:35.812476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.812457    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.821783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.821752    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:07:35.843474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.843450    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4gb\" (UniqueName: \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.843612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.843489    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.866905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.866882    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qkljv\""
Oct 31 16:07:35.944558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.944532    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.944713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.944662    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4gb\" (UniqueName: \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.944968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.944951    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:35.999099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:35.999078    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4gb\" (UniqueName: \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\") pod \"kuadrant-operator-controller-manager-678d87766f-qzfn6\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:36.124839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:36.124779    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:36.622527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:36.622490    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:07:36.641515 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:07:36.641486    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10d41d0_e0d5_44ee_8c48_61aae8a54225.slice/crio-84fb545db50a2d6885a5751d285db2bf2f9deb7272bf949264ac88cc4d73611e WatchSource:0}: Error finding container 84fb545db50a2d6885a5751d285db2bf2f9deb7272bf949264ac88cc4d73611e: Status 404 returned error can't find the container with id 84fb545db50a2d6885a5751d285db2bf2f9deb7272bf949264ac88cc4d73611e
Oct 31 16:07:37.154210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:37.154172    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" event={"ID":"e10d41d0-e0d5-44ee-8c48-61aae8a54225","Type":"ContainerStarted","Data":"84fb545db50a2d6885a5751d285db2bf2f9deb7272bf949264ac88cc4d73611e"}
Oct 31 16:07:41.174746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:41.174710    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w" event={"ID":"138006b7-3c26-4668-b8ab-eb13c70406a2","Type":"ContainerStarted","Data":"06295d9a885067642d72d146fdb76285aae670fde6665cfeffd05336865c2d6b"}
Oct 31 16:07:41.281318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:41.281263    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c4h8w" podStartSLOduration=1.6626574939999998 podStartE2EDuration="8.281247872s" totalImagesPullingTime="6.618590378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:41.27989868 +0000 UTC m=+306.341473866" watchObservedRunningTime="2026-10-31 16:07:41.281247872 +0000 UTC m=+306.342823057"
Oct 31 16:07:42.180429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:42.180393    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" event={"ID":"e10d41d0-e0d5-44ee-8c48-61aae8a54225","Type":"ContainerStarted","Data":"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"}
Oct 31 16:07:42.180914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:42.180490    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:07:42.382164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:42.382111    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" podStartSLOduration=2.055116533 podStartE2EDuration="7.382094267s" totalImagesPullingTime="5.326977734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:42.33477799 +0000 UTC m=+307.396353173" watchObservedRunningTime="2026-10-31 16:07:42.382094267 +0000 UTC m=+307.443669452"
Oct 31 16:07:53.186662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:07:53.186564    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:08:02.943484 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:08:02.943444    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:08:32.942443 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:08:32.942403    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:08:40.043906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.043869    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:08:40.046337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.046321    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082208    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082246    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082263    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082323    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8xm4\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082442    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082482    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.082656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.082541    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.142191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.142164    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:08:40.183539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183505    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183542    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183673    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183706    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183750    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8xm4\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183771    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.183995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.183803    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.184719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.184681    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.186148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.186121    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.186148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.186142    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.186297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.186197    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.186396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.186378    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.243976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.243941    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.244099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.244080    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8xm4\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ndttp\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.356885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.356823    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:40.782237 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:08:40.782207    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a0df2e_35fe_4c53_90a8_fb3e18d60366.slice/crio-55d4e788e8b13a99e371612b582b5ca0fc58058725cab03bef800c8f2896ac2a WatchSource:0}: Error finding container 55d4e788e8b13a99e371612b582b5ca0fc58058725cab03bef800c8f2896ac2a: Status 404 returned error can't find the container with id 55d4e788e8b13a99e371612b582b5ca0fc58058725cab03bef800c8f2896ac2a
Oct 31 16:08:40.784362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.784325    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:08:40.784438 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.784391    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:08:40.784895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:40.784812    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:08:41.407920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.407883    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" event={"ID":"84a0df2e-35fe-4c53-90a8-fb3e18d60366","Type":"ContainerStarted","Data":"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"}
Oct 31 16:08:41.408317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.407940    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" event={"ID":"84a0df2e-35fe-4c53-90a8-fb3e18d60366","Type":"ContainerStarted","Data":"55d4e788e8b13a99e371612b582b5ca0fc58058725cab03bef800c8f2896ac2a"}
Oct 31 16:08:41.408317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.407966    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:41.409566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.409532    4417 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ndttp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:08:41.409717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.409584    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:08:41.572563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:41.572512    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" podStartSLOduration=2.5724995059999998 podStartE2EDuration="2.572499506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:41.530931926 +0000 UTC m=+366.592507112" watchObservedRunningTime="2026-10-31 16:08:41.572499506 +0000 UTC m=+366.634074692"
Oct 31 16:08:42.412138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.412110    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:08:42.648471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.648433    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:08:42.648670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.648649    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery" containerID="cri-o://fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584" gracePeriod=30
Oct 31 16:08:42.804691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.804654    4417 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mhsg7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused" start-of-body=
Oct 31 16:08:42.804824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.804719    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.18:8080/ready\": dial tcp 10.133.0.18:8080: connect: connection refused"
Oct 31 16:08:42.884577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.884551    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:08:42.907252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907231    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907281    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907318    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907370    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907390    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907406    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907448    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\") pod \"e01370d6-1248-43b5-b53e-b28d32750cd0\" (UID: \"e01370d6-1248-43b5-b53e-b28d32750cd0\") "
Oct 31 16:08:42.907698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.907662    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:08:42.909612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.909582    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:42.909612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.909646    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:08:42.909946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.909917    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "kube-api-access-mq89c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:08:42.910052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.909949    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:42.910052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.909998    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:08:42.910052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:42.910013    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert" pod "e01370d6-1248-43b5-b53e-b28d32750cd0" (UID: "e01370d6-1248-43b5-b53e-b28d32750cd0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008735    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-ca-configmap\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008762    4417 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-kubeconfig\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008777    4417 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01370d6-1248-43b5-b53e-b28d32750cd0-local-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008786    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq89c\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-kube-api-access-mq89c\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008795    4417 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-cacerts\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.008804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008803    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.009073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.008811    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e01370d6-1248-43b5-b53e-b28d32750cd0-istio-csr-dns-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:08:43.416134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.416100    4417 generic.go:356] "Generic (PLEG): container finished" podID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerID="fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584" exitCode=0
Oct 31 16:08:43.416543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.416150    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"
Oct 31 16:08:43.416543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.416183    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" event={"ID":"e01370d6-1248-43b5-b53e-b28d32750cd0","Type":"ContainerDied","Data":"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"}
Oct 31 16:08:43.416543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.416241    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7" event={"ID":"e01370d6-1248-43b5-b53e-b28d32750cd0","Type":"ContainerDied","Data":"27c82b5e395a187d4fe170fe4cb60b7fed2646842500e579ace149c3945caf48"}
Oct 31 16:08:43.416543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.416262    4417 scope.go:122] "RemoveContainer" containerID="fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"
Oct 31 16:08:43.424549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.424531    4417 scope.go:122] "RemoveContainer" containerID="fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"
Oct 31 16:08:43.424804 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:08:43.424784    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584\": container with ID starting with fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584 not found: ID does not exist" containerID="fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"
Oct 31 16:08:43.424872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.424815    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584"} err="failed to get container status \"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584\": rpc error: code = NotFound desc = could not find container \"fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584\": container with ID starting with fbc94e014a86938f697a9e098fb274ef2b7686b31ca6f19010620a194124f584 not found: ID does not exist"
Oct 31 16:08:43.590681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.590653    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:08:43.644151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:43.644128    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mhsg7"]
Oct 31 16:08:45.047076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:45.047044    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" path="/var/lib/kubelet/pods/e01370d6-1248-43b5-b53e-b28d32750cd0/volumes"
Oct 31 16:08:47.503743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.503712    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m4nbt"]
Oct 31 16:08:47.504101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.503980    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery"
Oct 31 16:08:47.504101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.503990    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery"
Oct 31 16:08:47.504101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.504059    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01370d6-1248-43b5-b53e-b28d32750cd0" containerName="discovery"
Oct 31 16:08:47.506024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.506005    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.545279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.545255    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-tls-certs\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.545385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.545294    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjks2\" (UniqueName: \"kubernetes.io/projected/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-kube-api-access-kjks2\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.561049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.561022    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m4nbt"]
Oct 31 16:08:47.612893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.612869    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 16:08:47.615182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.615160    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 16:08:47.615266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.615164    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rp29v\""
Oct 31 16:08:47.615266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.615193    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 16:08:47.646071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.646048    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjks2\" (UniqueName: \"kubernetes.io/projected/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-kube-api-access-kjks2\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.646164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.646124    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-tls-certs\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.648472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.648450    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-tls-certs\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.705959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.705932    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjks2\" (UniqueName: \"kubernetes.io/projected/c73e7d4e-3bee-4d9c-af5a-7f63e560806a-kube-api-access-kjks2\") pod \"model-serving-api-75bd89fdfc-m4nbt\" (UID: \"c73e7d4e-3bee-4d9c-af5a-7f63e560806a\") " pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:47.816821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:47.816798    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:48.187660 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:08:48.187610    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73e7d4e_3bee_4d9c_af5a_7f63e560806a.slice/crio-6499a2ff634def2454809e8b5ebf627a243d9a19fcbacf6f3c12eaecc4ef9118 WatchSource:0}: Error finding container 6499a2ff634def2454809e8b5ebf627a243d9a19fcbacf6f3c12eaecc4ef9118: Status 404 returned error can't find the container with id 6499a2ff634def2454809e8b5ebf627a243d9a19fcbacf6f3c12eaecc4ef9118
Oct 31 16:08:48.189746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:48.189723    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:08:48.190416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:48.190378    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-m4nbt"]
Oct 31 16:08:48.438277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:48.438197    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt" event={"ID":"c73e7d4e-3bee-4d9c-af5a-7f63e560806a","Type":"ContainerStarted","Data":"6499a2ff634def2454809e8b5ebf627a243d9a19fcbacf6f3c12eaecc4ef9118"}
Oct 31 16:08:51.452474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:51.452432    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt" event={"ID":"c73e7d4e-3bee-4d9c-af5a-7f63e560806a","Type":"ContainerStarted","Data":"24142c3fae5b4eac6c32df9f24202faa10b58e284dac245fa52c4e9acaba4f89"}
Oct 31 16:08:51.452952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:51.452561    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:08:51.572982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:08:51.572935    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt" podStartSLOduration=2.258586438 podStartE2EDuration="4.572920936s" totalImagesPullingTime="2.314334498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:51.571840459 +0000 UTC m=+376.633415645" watchObservedRunningTime="2026-10-31 16:08:51.572920936 +0000 UTC m=+376.634496122"
Oct 31 16:09:00.317239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.317207    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:00.320174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.320152    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.375165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.375131    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:00.427847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.427811    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:09:00.429517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.429494    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:09:00.429517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.429511    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 16:09:00.429684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.429562    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\""
Oct 31 16:09:00.429684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.429550    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:09:00.447907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.447874    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.447907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.447906    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.448084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.447928    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.448084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.448014    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6lm\" (UniqueName: \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.549228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.549187    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.549228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.549228    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.549464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.549248    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.549464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.549272    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6lm\" (UniqueName: \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.550089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.550056    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.551724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.551699    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.551833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.551772    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.609999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.609927    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6lm\" (UniqueName: \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\") pod \"maas-api-6d669c7fcd-ftb9x\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:00.632816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:00.632779    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:01.005680 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:09:01.005651    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ea44e4_5ad7_4276_a729_fb6c0ae770eb.slice/crio-a1368f16b52f3788027e986a54765c7e791adb437a12a6cc9c709858de259328 WatchSource:0}: Error finding container a1368f16b52f3788027e986a54765c7e791adb437a12a6cc9c709858de259328: Status 404 returned error can't find the container with id a1368f16b52f3788027e986a54765c7e791adb437a12a6cc9c709858de259328
Oct 31 16:09:01.008527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:01.008506    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:01.492043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:01.492002    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" event={"ID":"25ea44e4-5ad7-4276-a729-fb6c0ae770eb","Type":"ContainerStarted","Data":"a1368f16b52f3788027e986a54765c7e791adb437a12a6cc9c709858de259328"}
Oct 31 16:09:02.460900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:02.460872    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt"
Oct 31 16:09:02.948659 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:09:02.948606    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:09:05.512271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:05.512230    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" event={"ID":"25ea44e4-5ad7-4276-a729-fb6c0ae770eb","Type":"ContainerStarted","Data":"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"}
Oct 31 16:09:05.512819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:05.512354    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:05.634016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:05.633966    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" podStartSLOduration=1.857994653 podStartE2EDuration="5.633949526s" totalImagesPullingTime="3.775954873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:05.632967484 +0000 UTC m=+390.694542670" watchObservedRunningTime="2026-10-31 16:09:05.633949526 +0000 UTC m=+390.695524708"
Oct 31 16:09:11.521366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:11.521331    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:23.020133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.020044    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:23.020683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.020292    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" podUID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" containerName="maas-api" containerID="cri-o://0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7" gracePeriod=30
Oct 31 16:09:23.259084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.259059    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:23.353421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.353386    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\") pod \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") "
Oct 31 16:09:23.353645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.353430    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\" (UniqueName: \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\") pod \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") "
Oct 31 16:09:23.353645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.353456    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\") pod \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") "
Oct 31 16:09:23.353932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.353904    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca" pod "25ea44e4-5ad7-4276-a729-fb6c0ae770eb" (UID: "25ea44e4-5ad7-4276-a729-fb6c0ae770eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:23.355781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.355748    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls" pod "25ea44e4-5ad7-4276-a729-fb6c0ae770eb" (UID: "25ea44e4-5ad7-4276-a729-fb6c0ae770eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:09:23.355781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.355753    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm" pod "25ea44e4-5ad7-4276-a729-fb6c0ae770eb" (UID: "25ea44e4-5ad7-4276-a729-fb6c0ae770eb"). InnerVolumeSpecName "kube-api-access-4q6lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:23.454165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.454133    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\") pod \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\" (UID: \"25ea44e4-5ad7-4276-a729-fb6c0ae770eb\") "
Oct 31 16:09:23.454342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.454292    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:09:23.454342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.454306    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6lm\" (UniqueName: \"kubernetes.io/projected/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-kube-api-access-4q6lm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:09:23.454342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.454317    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:09:23.456143 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.456120    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs" pod "25ea44e4-5ad7-4276-a729-fb6c0ae770eb" (UID: "25ea44e4-5ad7-4276-a729-fb6c0ae770eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:09:23.555440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.555396    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ea44e4-5ad7-4276-a729-fb6c0ae770eb-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:09:23.585366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.585333    4417 generic.go:356] "Generic (PLEG): container finished" podID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" containerID="0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7" exitCode=0
Oct 31 16:09:23.585550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.585390    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"
Oct 31 16:09:23.585550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.585408    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" event={"ID":"25ea44e4-5ad7-4276-a729-fb6c0ae770eb","Type":"ContainerDied","Data":"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"}
Oct 31 16:09:23.585550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.585460    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x" event={"ID":"25ea44e4-5ad7-4276-a729-fb6c0ae770eb","Type":"ContainerDied","Data":"a1368f16b52f3788027e986a54765c7e791adb437a12a6cc9c709858de259328"}
Oct 31 16:09:23.585550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.585482    4417 scope.go:122] "RemoveContainer" containerID="0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"
Oct 31 16:09:23.594285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.594267    4417 scope.go:122] "RemoveContainer" containerID="0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"
Oct 31 16:09:23.594598 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:09:23.594576    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7\": container with ID starting with 0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7 not found: ID does not exist" containerID="0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"
Oct 31 16:09:23.594670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.594626    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7"} err="failed to get container status \"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7\": rpc error: code = NotFound desc = could not find container \"0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7\": container with ID starting with 0f7a8a3bb12d269fe991c631be840d53c484586c3cf3082b4755f67610c441a7 not found: ID does not exist"
Oct 31 16:09:23.759127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.759044    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:23.812827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:23.812787    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ftb9x"]
Oct 31 16:09:25.047271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:09:25.047229    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" path="/var/lib/kubelet/pods/25ea44e4-5ad7-4276-a729-fb6c0ae770eb/volumes"
Oct 31 16:09:32.943456 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:09:32.943421    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:10:02.943214 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:10:02.943174    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:10:32.943101 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:10:32.943064    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:10:50.683295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.683218    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"]
Oct 31 16:10:50.683726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.683553    4417 cpu_manager.go:414] "containerMap: removing container" podUID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" containerName="maas-api"
Oct 31 16:10:50.683726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.683564    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" containerName="maas-api"
Oct 31 16:10:50.683726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.683630    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ea44e4-5ad7-4276-a729-fb6c0ae770eb" containerName="maas-api"
Oct 31 16:10:50.686468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.686448    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740201    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59fac8e-9759-44e0-9ace-6d21585d03d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740237    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740263    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6rt\" (UniqueName: \"kubernetes.io/projected/d59fac8e-9759-44e0-9ace-6d21585d03d3-kube-api-access-tx6rt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740337    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740381    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.740560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.740446    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.780604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.780559    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 16:10:50.780824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.780667    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ngqjx\""
Oct 31 16:10:50.780824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.780707    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 16:10:50.780824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.780745    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 16:10:50.782251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.782227    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"]
Oct 31 16:10:50.841145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841111    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841174    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841209    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59fac8e-9759-44e0-9ace-6d21585d03d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841239    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841264    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6rt\" (UniqueName: \"kubernetes.io/projected/d59fac8e-9759-44e0-9ace-6d21585d03d3-kube-api-access-tx6rt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841296    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841544    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841598    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.841823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.841797    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.843478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.843455    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59fac8e-9759-44e0-9ace-6d21585d03d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.843641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.843603    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59fac8e-9759-44e0-9ace-6d21585d03d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.894008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.893979    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6rt\" (UniqueName: \"kubernetes.io/projected/d59fac8e-9759-44e0-9ace-6d21585d03d3-kube-api-access-tx6rt\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh\" (UID: \"d59fac8e-9759-44e0-9ace-6d21585d03d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:50.998534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:50.998448    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:51.366482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:51.366444    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"]
Oct 31 16:10:51.373842 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:10:51.373807    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59fac8e_9759_44e0_9ace_6d21585d03d3.slice/crio-03d856163f4708a16d85e90a4b35b7f9c2ebe9bd37e9aa49f76dccb42b6b4b48 WatchSource:0}: Error finding container 03d856163f4708a16d85e90a4b35b7f9c2ebe9bd37e9aa49f76dccb42b6b4b48: Status 404 returned error can't find the container with id 03d856163f4708a16d85e90a4b35b7f9c2ebe9bd37e9aa49f76dccb42b6b4b48
Oct 31 16:10:51.932294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:51.932253    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh" event={"ID":"d59fac8e-9759-44e0-9ace-6d21585d03d3","Type":"ContainerStarted","Data":"03d856163f4708a16d85e90a4b35b7f9c2ebe9bd37e9aa49f76dccb42b6b4b48"}
Oct 31 16:10:53.944135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:53.944098    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh" event={"ID":"d59fac8e-9759-44e0-9ace-6d21585d03d3","Type":"ContainerStarted","Data":"b4897057da43854df68eed47fe03caf47e1681a1f11afa02b332c7ec4018aad9"}
Oct 31 16:10:53.998580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:53.998547    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:10:54.050822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.050771    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh" podStartSLOduration=2.033734708 podStartE2EDuration="4.050759164s" totalImagesPullingTime="2.017024456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:54.04972193 +0000 UTC m=+499.111297117" watchObservedRunningTime="2026-10-31 16:10:54.050759164 +0000 UTC m=+499.112334350"
Oct 31 16:10:54.512560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.512522    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"]
Oct 31 16:10:54.515897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.515876    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.564195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.564166    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 16:10:54.574819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.574788    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.575019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.574838    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.575019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.574921    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqnn\" (UniqueName: \"kubernetes.io/projected/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-kube-api-access-dwqnn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.575019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.574981    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.575019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.575011    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.575195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.575079    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.611061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.611027    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"]
Oct 31 16:10:54.675972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.675933    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676007    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqnn\" (UniqueName: \"kubernetes.io/projected/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-kube-api-access-dwqnn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676039    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676063    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676112    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676136    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676517    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676544    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.676657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.676576    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.678406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.678381    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.678503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.678425    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.728689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.728658    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqnn\" (UniqueName: \"kubernetes.io/projected/71f2171c-108c-4ae4-aad3-d0bc9cf1cd61-kube-api-access-dwqnn\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-f98mh\" (UID: \"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:54.828550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:54.828516    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:10:55.191423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:55.191342    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"]
Oct 31 16:10:55.195118 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:10:55.195079    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f2171c_108c_4ae4_aad3_d0bc9cf1cd61.slice/crio-d7905e1e89f4bec7350b8e4f89348eeecd57fa992e0c44e39a97598dfd1aaa48 WatchSource:0}: Error finding container d7905e1e89f4bec7350b8e4f89348eeecd57fa992e0c44e39a97598dfd1aaa48: Status 404 returned error can't find the container with id d7905e1e89f4bec7350b8e4f89348eeecd57fa992e0c44e39a97598dfd1aaa48
Oct 31 16:10:55.954568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:55.954513    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh" event={"ID":"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61","Type":"ContainerStarted","Data":"580a572493856f0e387bfe05f1dd491ada232547d55ed40c177c7a08a4c0a0d2"}
Oct 31 16:10:55.954904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:55.954591    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh" event={"ID":"71f2171c-108c-4ae4-aad3-d0bc9cf1cd61","Type":"ContainerStarted","Data":"d7905e1e89f4bec7350b8e4f89348eeecd57fa992e0c44e39a97598dfd1aaa48"}
Oct 31 16:10:56.061458 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:56.061408    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh" podStartSLOduration=1.481657083 podStartE2EDuration="2.061392732s" totalImagesPullingTime="579.735649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:56.061113333 +0000 UTC m=+501.122688530" watchObservedRunningTime="2026-10-31 16:10:56.061392732 +0000 UTC m=+501.122967919"
Oct 31 16:10:56.828955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:10:56.828915    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:11:00.999179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:00.999145    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:11:01.011500 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.011475    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:11:01.554465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.554429    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"]
Oct 31 16:11:01.558025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.558003    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.606265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.606237    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 16:11:01.615280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.615256    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"]
Oct 31 16:11:01.635375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635345    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.635531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.635531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635413    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5csg\" (UniqueName: \"kubernetes.io/projected/eb823ecd-bf57-48c0-a47a-9b0536b6c531-kube-api-access-v5csg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.635531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635502    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.635531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635517    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.635721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.635591    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736424    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736465    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5csg\" (UniqueName: \"kubernetes.io/projected/eb823ecd-bf57-48c0-a47a-9b0536b6c531-kube-api-access-v5csg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736490    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736609    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736698    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736789    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.736958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736923    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.737006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.736970    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.737133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.737109    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.738855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.738834    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb823ecd-bf57-48c0-a47a-9b0536b6c531-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.739185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.739167    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb823ecd-bf57-48c0-a47a-9b0536b6c531-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.792562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.792522    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5csg\" (UniqueName: \"kubernetes.io/projected/eb823ecd-bf57-48c0-a47a-9b0536b6c531-kube-api-access-v5csg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-89jt4\" (UID: \"eb823ecd-bf57-48c0-a47a-9b0536b6c531\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.871828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.871734    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:01.991634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:01.991587    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh"
Oct 31 16:11:02.242869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:02.242778    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"]
Oct 31 16:11:02.247638 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:02.247579    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb823ecd_bf57_48c0_a47a_9b0536b6c531.slice/crio-032c435f5dbbc374067047a2b1c3fc0685f300b9c4ac519aa3c4cb0e113be357 WatchSource:0}: Error finding container 032c435f5dbbc374067047a2b1c3fc0685f300b9c4ac519aa3c4cb0e113be357: Status 404 returned error can't find the container with id 032c435f5dbbc374067047a2b1c3fc0685f300b9c4ac519aa3c4cb0e113be357
Oct 31 16:11:02.947244 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:11:02.947206    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:11:02.986770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:02.986726    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4" event={"ID":"eb823ecd-bf57-48c0-a47a-9b0536b6c531","Type":"ContainerStarted","Data":"2812fd4e28a18506acc52ba763bc292a8ce6fbacad1821fb30a8740b4a999ca5"}
Oct 31 16:11:02.986966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:02.986790    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4" event={"ID":"eb823ecd-bf57-48c0-a47a-9b0536b6c531","Type":"ContainerStarted","Data":"032c435f5dbbc374067047a2b1c3fc0685f300b9c4ac519aa3c4cb0e113be357"}
Oct 31 16:11:03.093258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:03.093206    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4" podStartSLOduration=1.496064158 podStartE2EDuration="2.093190357s" totalImagesPullingTime="597.126199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:03.091485643 +0000 UTC m=+508.153060829" watchObservedRunningTime="2026-10-31 16:11:03.093190357 +0000 UTC m=+508.154765543"
Oct 31 16:11:03.872254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:03.872211    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:04.266611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.266519    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"]
Oct 31 16:11:04.270001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.269974    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361584    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fdn\" (UniqueName: \"kubernetes.io/projected/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-kube-api-access-v5fdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361647    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361741    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361794    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361907    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.361953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.361931    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.363790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.363773    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 16:11:04.365157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.365139    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"]
Oct 31 16:11:04.462385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462341    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462432    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462459    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462500    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fdn\" (UniqueName: \"kubernetes.io/projected/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-kube-api-access-v5fdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462665    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462723    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462801    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.462914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.462859    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.463054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.463027    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.464772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.464746    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.465065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.465046    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.515249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.515206    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fdn\" (UniqueName: \"kubernetes.io/projected/cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc-kube-api-access-v5fdn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p\" (UID: \"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.582905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.582865    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:04.828941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.828906    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:11:04.841831 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.841763    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:11:04.950534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.950498    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"]
Oct 31 16:11:04.957638 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:04.957591    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6afbca_dbe9_4895_ad67_7c55c0d3f6bc.slice/crio-b4bee1b850f6f28a3ae0fe61305ea5969bac626691df7969224e946cde08a6c7 WatchSource:0}: Error finding container b4bee1b850f6f28a3ae0fe61305ea5969bac626691df7969224e946cde08a6c7: Status 404 returned error can't find the container with id b4bee1b850f6f28a3ae0fe61305ea5969bac626691df7969224e946cde08a6c7
Oct 31 16:11:04.995910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:04.995876    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p" event={"ID":"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc","Type":"ContainerStarted","Data":"b4bee1b850f6f28a3ae0fe61305ea5969bac626691df7969224e946cde08a6c7"}
Oct 31 16:11:05.007544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:05.007516    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh"
Oct 31 16:11:06.001991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:06.001956    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p" event={"ID":"cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc","Type":"ContainerStarted","Data":"3df24fd5aa4aacaae645256856e177477a3731d75d37749b81c2dc1b4cbd4bf0"}
Oct 31 16:11:06.109498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:06.109441    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p" podStartSLOduration=1.580344909 podStartE2EDuration="2.109425464s" totalImagesPullingTime="529.080555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:06.107369255 +0000 UTC m=+511.168944423" watchObservedRunningTime="2026-10-31 16:11:06.109425464 +0000 UTC m=+511.171000649"
Oct 31 16:11:06.583197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:06.583163    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:11.872214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:11.872172    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:11.884754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:11.884724    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:12.042972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:12.042943    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-89jt4"
Oct 31 16:11:14.583185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:14.583152    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:14.595539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:14.595514    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:15.055643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:15.055593    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p"
Oct 31 16:11:32.942979 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:11:32.942939    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:11:34.896537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.896500    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:11:34.901660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.901607    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932066    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932099    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932120    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932135    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlh99\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932268    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932302    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932329    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932376    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.932596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.932475    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:34.942438 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.942414    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:11:34.945813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.945798    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:34.955826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:34.955807    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gw56j\""
Oct 31 16:11:35.000596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.000568    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:11:35.032924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.032893    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.032924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.032927    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.032949    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.032965    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033096    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dnj\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033160    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033185    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033203    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033220    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033355    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033384    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033392    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033470    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033500    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.033536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033533    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033549    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlh99\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033607    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033675    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033712    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.033840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.033716    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.034093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.034070    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.034298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.034267    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.034375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.034334    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.036089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.036069    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.036243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.036060    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.050683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.050660    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:11:35.085027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.084995    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.085364 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.085346    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlh99\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7c9qm\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.134441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134406    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134448    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134474    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dnj\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134514    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134533    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134579    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134604    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134673    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134695    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134888    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.134952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134934    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.135165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.134981    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.135165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.135076    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.135452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.135430    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.137064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.137038    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.137302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.137284    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.187105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.187027    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.187254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.187166    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dnj\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cnft9\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.260465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.260433    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gw56j\""
Oct 31 16:11:35.263503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.263481    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:35.268594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.268575    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:35.684962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.684923    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:11:35.688001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:35.687967    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:11:35.692512 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:35.692488    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15efc78_8629_4ca3_9f40_dca9ebfb7dcc.slice/crio-4d28f1668a5b723a867e0b431202f51bda2bec5689f43ad0a96a623fc3e06cf7 WatchSource:0}: Error finding container 4d28f1668a5b723a867e0b431202f51bda2bec5689f43ad0a96a623fc3e06cf7: Status 404 returned error can't find the container with id 4d28f1668a5b723a867e0b431202f51bda2bec5689f43ad0a96a623fc3e06cf7
Oct 31 16:11:35.694990 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:35.694968    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3de8cc_4e7a_491d_99ba_b85e85b9e2b5.slice/crio-d44c2236d0a5a3768bd81ec798d2dfc28225317b511518565e1c6506a6ab202a WatchSource:0}: Error finding container d44c2236d0a5a3768bd81ec798d2dfc28225317b511518565e1c6506a6ab202a: Status 404 returned error can't find the container with id d44c2236d0a5a3768bd81ec798d2dfc28225317b511518565e1c6506a6ab202a
Oct 31 16:11:36.130816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:36.130775    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" event={"ID":"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5","Type":"ContainerStarted","Data":"d44c2236d0a5a3768bd81ec798d2dfc28225317b511518565e1c6506a6ab202a"}
Oct 31 16:11:36.131889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:36.131866    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" event={"ID":"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc","Type":"ContainerStarted","Data":"4d28f1668a5b723a867e0b431202f51bda2bec5689f43ad0a96a623fc3e06cf7"}
Oct 31 16:11:40.066044 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.066008    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.066434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.066081    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.066434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.066113    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.074910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.074880    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.075032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.074946    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.075032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.074975    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:11:40.153344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.153307    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" event={"ID":"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc","Type":"ContainerStarted","Data":"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"}
Oct 31 16:11:40.155260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.155227    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" event={"ID":"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5","Type":"ContainerStarted","Data":"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"}
Oct 31 16:11:40.264428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.264396    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:40.269801 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.269774    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:40.273612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.273574    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podStartSLOduration=1.9031017879999998 podStartE2EDuration="6.273562138s" totalImagesPullingTime="4.37046035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:40.272515005 +0000 UTC m=+545.334090203" watchObservedRunningTime="2026-10-31 16:11:40.273562138 +0000 UTC m=+545.335137323"
Oct 31 16:11:40.380387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.380265    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podStartSLOduration=2.002838569 podStartE2EDuration="6.380245654s" totalImagesPullingTime="4.377407085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:40.378860363 +0000 UTC m=+545.440435549" watchObservedRunningTime="2026-10-31 16:11:40.380245654 +0000 UTC m=+545.441820842"
Oct 31 16:11:40.596694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.596644    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:11:40.601322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.601290    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.683782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.658649    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:11:40.683782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.659352    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 16:11:40.683782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.659416    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 16:11:40.683782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.659548    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5s6jt\""
Oct 31 16:11:40.696641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.696381    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.696641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.696428    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx468\" (UniqueName: \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.797347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.797303    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.797347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.797347    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx468\" (UniqueName: \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.799814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.799786    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.859646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.856350    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx468\" (UniqueName: \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\") pod \"ai-gateway-controller-5f6dd4f69f-6n9s9\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:40.921338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:40.921295    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:41.283722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:41.283427    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:41.283722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:41.283495    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:41.284841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:41.284814    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:41.284963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:41.284866    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:41.388022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:41.387982    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:11:41.391216 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:41.390214    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be12911_b738_4849_a912_2c1baf630991.slice/crio-05f88541b245da0636871f4d811b4260b906f9191673d46d1681dda0f9ee7faa WatchSource:0}: Error finding container 05f88541b245da0636871f4d811b4260b906f9191673d46d1681dda0f9ee7faa: Status 404 returned error can't find the container with id 05f88541b245da0636871f4d811b4260b906f9191673d46d1681dda0f9ee7faa
Oct 31 16:11:42.209233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:42.209197    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" event={"ID":"8be12911-b738-4849-a912-2c1baf630991","Type":"ContainerStarted","Data":"05f88541b245da0636871f4d811b4260b906f9191673d46d1681dda0f9ee7faa"}
Oct 31 16:11:42.264058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:42.264017    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:42.264331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:42.264301    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:42.273013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:42.272976    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:42.273267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:42.273235    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:43.268093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:43.268052    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:43.268693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:43.268654    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:43.269479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:43.269455    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:43.269709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:43.269685    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:44.264051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:44.264011    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:44.264356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:44.264323    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:44.283040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:44.283002    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:44.283709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:44.283675    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:45.273649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:45.272901    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:45.273649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:45.272960    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:45.273649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:45.273362    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:45.273649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:45.273405    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:46.264086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:46.264047    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:46.264566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:46.264111    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:46.269331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:46.269307    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:46.269408 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:46.269359    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:47.264816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:47.264770    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7c9qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 31 16:11:47.265334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:47.264847    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 31 16:11:47.269172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:47.269139    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 31 16:11:47.269322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:47.269207    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 31 16:11:48.243056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.242997    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" event={"ID":"8be12911-b738-4849-a912-2c1baf630991","Type":"ContainerStarted","Data":"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"}
Oct 31 16:11:48.243441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.243414    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:11:48.354018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.353927    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" podStartSLOduration=2.327337889 podStartE2EDuration="8.353907508s" totalImagesPullingTime="6.026569619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:48.352545685 +0000 UTC m=+553.414120872" watchObservedRunningTime="2026-10-31 16:11:48.353907508 +0000 UTC m=+553.415482694"
Oct 31 16:11:48.752683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.751268    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:48.752683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.751783    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:48.752872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.752769    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:11:48.784360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.784320    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:48.784763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.784736    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:48.785908 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:48.785887    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:11:50.807721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.807678    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"]
Oct 31 16:11:50.812079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.812052    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:50.906339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.906298    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 16:11:50.906339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.906300    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b4pgc\""
Oct 31 16:11:50.907919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.907892    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"]
Oct 31 16:11:50.968974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.968929    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b03b4c-9d0a-41c0-8f6d-171923978a8c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:50.968974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:50.968976    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crwj\" (UniqueName: \"kubernetes.io/projected/49b03b4c-9d0a-41c0-8f6d-171923978a8c-kube-api-access-5crwj\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.070402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.070300    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crwj\" (UniqueName: \"kubernetes.io/projected/49b03b4c-9d0a-41c0-8f6d-171923978a8c-kube-api-access-5crwj\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.070582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.070452    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b03b4c-9d0a-41c0-8f6d-171923978a8c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.071136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.071110    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b03b4c-9d0a-41c0-8f6d-171923978a8c-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.123173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.123127    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crwj\" (UniqueName: \"kubernetes.io/projected/49b03b4c-9d0a-41c0-8f6d-171923978a8c-kube-api-access-5crwj\") pod \"payload-processing-55fbbbf9dc-9f4p2\" (UID: \"49b03b4c-9d0a-41c0-8f6d-171923978a8c\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.127747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.127719    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:51.500965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:51.500868    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"]
Oct 31 16:11:51.509879 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:11:51.509845    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b03b4c_9d0a_41c0_8f6d_171923978a8c.slice/crio-6f6f82118ea0a0e577c79812c10a6344a557f2c8e15b6fd1fa8f682e4f08e8de WatchSource:0}: Error finding container 6f6f82118ea0a0e577c79812c10a6344a557f2c8e15b6fd1fa8f682e4f08e8de: Status 404 returned error can't find the container with id 6f6f82118ea0a0e577c79812c10a6344a557f2c8e15b6fd1fa8f682e4f08e8de
Oct 31 16:11:52.265991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:52.265950    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2" event={"ID":"49b03b4c-9d0a-41c0-8f6d-171923978a8c","Type":"ContainerStarted","Data":"6f6f82118ea0a0e577c79812c10a6344a557f2c8e15b6fd1fa8f682e4f08e8de"}
Oct 31 16:11:55.291187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:55.291142    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2" event={"ID":"49b03b4c-9d0a-41c0-8f6d-171923978a8c","Type":"ContainerStarted","Data":"4c64e9730bd88ec5aee8686fa9263a26dcd895108129c267ca6965f9cb9d03f9"}
Oct 31 16:11:55.291604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:55.291288    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:11:55.396340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:55.396287    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2" podStartSLOduration=2.358661654 podStartE2EDuration="5.396272653s" totalImagesPullingTime="3.037610999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:55.39538288 +0000 UTC m=+560.456958066" watchObservedRunningTime="2026-10-31 16:11:55.396272653 +0000 UTC m=+560.457847839"
Oct 31 16:11:59.250158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:11:59.250121    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:12:02.944373 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:12:02.944338    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:12:06.299504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:06.299467    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9f4p2"
Oct 31 16:12:15.049853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.049819    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:12:15.050258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.050034    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" podUID="8be12911-b738-4849-a912-2c1baf630991" containerName="manager" containerID="cri-o://4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1" gracePeriod=30
Oct 31 16:12:15.293869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.293845    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:12:15.385752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.385669    4417 generic.go:356] "Generic (PLEG): container finished" podID="8be12911-b738-4849-a912-2c1baf630991" containerID="4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1" exitCode=0
Oct 31 16:12:15.385752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.385722    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" event={"ID":"8be12911-b738-4849-a912-2c1baf630991","Type":"ContainerDied","Data":"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"}
Oct 31 16:12:15.385995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.385759    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9" event={"ID":"8be12911-b738-4849-a912-2c1baf630991","Type":"ContainerDied","Data":"05f88541b245da0636871f4d811b4260b906f9191673d46d1681dda0f9ee7faa"}
Oct 31 16:12:15.385995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.385780    4417 scope.go:122] "RemoveContainer" containerID="4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"
Oct 31 16:12:15.385995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.385723    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"
Oct 31 16:12:15.393777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.393762    4417 scope.go:122] "RemoveContainer" containerID="4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"
Oct 31 16:12:15.394032 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:12:15.394010    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1\": container with ID starting with 4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1 not found: ID does not exist" containerID="4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"
Oct 31 16:12:15.394094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.394042    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1"} err="failed to get container status \"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1\": rpc error: code = NotFound desc = could not find container \"4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1\": container with ID starting with 4ab7361b36f5dac24dd584f8b65e80105872665057fac92807b8454d47f0c0c1 not found: ID does not exist"
Oct 31 16:12:15.401606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.401587    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\" (UniqueName: \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\") pod \"8be12911-b738-4849-a912-2c1baf630991\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") "
Oct 31 16:12:15.401725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.401627    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\") pod \"8be12911-b738-4849-a912-2c1baf630991\" (UID: \"8be12911-b738-4849-a912-2c1baf630991\") "
Oct 31 16:12:15.404232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.404206    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs" pod "8be12911-b738-4849-a912-2c1baf630991" (UID: "8be12911-b738-4849-a912-2c1baf630991"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:12:15.404355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.404331    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468" pod "8be12911-b738-4849-a912-2c1baf630991" (UID: "8be12911-b738-4849-a912-2c1baf630991"). InnerVolumeSpecName "kube-api-access-dx468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:12:15.502793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.502757    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx468\" (UniqueName: \"kubernetes.io/projected/8be12911-b738-4849-a912-2c1baf630991-kube-api-access-dx468\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:12:15.502793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.502787    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be12911-b738-4849-a912-2c1baf630991-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:12:15.796947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.796912    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:12:15.842642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:15.842591    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-6n9s9"]
Oct 31 16:12:17.046844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:17.046763    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be12911-b738-4849-a912-2c1baf630991" path="/var/lib/kubelet/pods/8be12911-b738-4849-a912-2c1baf630991/volumes"
Oct 31 16:12:32.943111 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:12:32.943073    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:12:45.244304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.244269    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:12:45.244849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.244750    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8be12911-b738-4849-a912-2c1baf630991" containerName="manager"
Oct 31 16:12:45.244849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.244768    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be12911-b738-4849-a912-2c1baf630991" containerName="manager"
Oct 31 16:12:45.244849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.244831    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be12911-b738-4849-a912-2c1baf630991" containerName="manager"
Oct 31 16:12:45.251607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.251586    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284410    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284468    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284499    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284540    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284564    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284599    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.284680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284661    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.285042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284700    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.285042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.284743    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j4n\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.312495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.312463    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe3a4-openshift-default-dockercfg-njqjl\""
Oct 31 16:12:45.348173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.348137    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:12:45.385410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385376    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385424    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385517    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385547    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385578    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385611    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385693    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j4n\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385741    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385778    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.385874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385840    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.386069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385950    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.386069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.385997    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.386204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.386165    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.386519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.386501    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.387726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.387703    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.388168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.388146    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.438466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.438427    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.438810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.438785    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j4n\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.564610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.564574    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:45.864156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.864069    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:12:45.868821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.868792    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.920461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.920427    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f7821a-openshift-default-dockercfg-pbwpd\""
Oct 31 16:12:45.923353 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.923029    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:12:45.992218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992186    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992240    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992312    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992332    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6sg5\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992352    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992428    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992509    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992566    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.992698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.992645    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:45.993148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:45.993127    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:12:45.999662 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:12:45.999637    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1baba935_5cd0_44bd_a815_f5742357b8b6.slice/crio-df685fe86f2587e5995598a0579e48eca5e5d1e0f92dd5a3b566071124d2bc47 WatchSource:0}: Error finding container df685fe86f2587e5995598a0579e48eca5e5d1e0f92dd5a3b566071124d2bc47: Status 404 returned error can't find the container with id df685fe86f2587e5995598a0579e48eca5e5d1e0f92dd5a3b566071124d2bc47
Oct 31 16:12:46.002202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.002171    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.002282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.002237    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.002282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.002267    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.093848    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.093914    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.093941    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6sg5\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.093970    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094018    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094058    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094105    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094138    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094221    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.094986    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.095266    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.095652    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.095901    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.095950    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.109243    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.109960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.109653    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.152819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.152738    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.152955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.152849    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6sg5\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.175992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.175958    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:12:46.180668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.180644    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.181795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.181775    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:46.229663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.228895    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ac1273-openshift-default-dockercfg-zfcwg\""
Oct 31 16:12:46.235249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.234730    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:12:46.295639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295570    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzs7p\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.295639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295609    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295655    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295820    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295858    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.295887    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.296018    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.296057    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.296257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.296098    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397336    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397421    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs7p\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397451    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397477    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397516    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397559    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397588    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397692    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.397932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.397727    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.398116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.398086    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.398177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.398096    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.398648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.398370    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.398648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.398545    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.398648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.398589    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.400392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.400367    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.400546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.400531    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.453332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.453254    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.453518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.453497    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzs7p\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.495767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.495731    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:46.530833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.530791    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerStarted","Data":"e2c8dfb191f63b0e66c666dbcb4562a55efe3acb0dda8f755f85a038fb836c23"}
Oct 31 16:12:46.531002 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.530865    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerStarted","Data":"df685fe86f2587e5995598a0579e48eca5e5d1e0f92dd5a3b566071124d2bc47"}
Oct 31 16:12:46.565311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.565280    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:46.584728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.584691    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:12:46.590530 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:12:46.590497    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80599a52_229a_4d25_a347_687580a623e8.slice/crio-9041f07d80b637cc23df2be88dcf59ed9668f11940b28e553e5c84dca343cbd0 WatchSource:0}: Error finding container 9041f07d80b637cc23df2be88dcf59ed9668f11940b28e553e5c84dca343cbd0: Status 404 returned error can't find the container with id 9041f07d80b637cc23df2be88dcf59ed9668f11940b28e553e5c84dca343cbd0
Oct 31 16:12:46.592928 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.592896    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.593001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.592959    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.593001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.592990    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.644448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.644391    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podStartSLOduration=1.64437155 podStartE2EDuration="1.64437155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:46.641880368 +0000 UTC m=+611.703455556" watchObservedRunningTime="2026-10-31 16:12:46.64437155 +0000 UTC m=+611.705946737"
Oct 31 16:12:46.873600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.873559    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:12:46.904118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.904057    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.904349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.904315    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:46.904503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:46.904485    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.147657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.147558    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:12:47.151154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.151130    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.214964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.214929    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ef6d93-openshift-default-dockercfg-dzsjw\""
Oct 31 16:12:47.220264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.220218    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:12:47.223099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.223068    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225087    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225243    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dcz\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225318    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225486    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225571    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225708    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.225749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225740    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.226104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.225820    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.326722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326689    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326740    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326763    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326797    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dcz\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326822    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326874    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326904    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326939    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.326964    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.327191    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.327240    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.327310    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.327479    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.327747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.327664    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.329203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.329183    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.329391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.329375    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.379345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.379313    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.379506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.379429    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dcz\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.467952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.467859    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:47.543982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.543932    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerStarted","Data":"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"}
Oct 31 16:12:47.544174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.544106    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerStarted","Data":"9041f07d80b637cc23df2be88dcf59ed9668f11940b28e553e5c84dca343cbd0"}
Oct 31 16:12:47.546859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.546829    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerStarted","Data":"6c0496446a5747a0565b44bce25c76af942f839c297f5949df9a20880c023923"}
Oct 31 16:12:47.546990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.546876    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerStarted","Data":"b13ae23efd08d4db3a9ef61f0bf93c1c0577b7a8f0895d5cd490d042e6ad121d"}
Oct 31 16:12:47.570754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.570722    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:47.657292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.657240    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podStartSLOduration=2.657225458 podStartE2EDuration="2.657225458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:47.655008469 +0000 UTC m=+612.716583656" watchObservedRunningTime="2026-10-31 16:12:47.657225458 +0000 UTC m=+612.718800644"
Oct 31 16:12:47.763216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.763117    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podStartSLOduration=1.76310329 podStartE2EDuration="1.76310329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:47.761088008 +0000 UTC m=+612.822663195" watchObservedRunningTime="2026-10-31 16:12:47.76310329 +0000 UTC m=+612.824678476"
Oct 31 16:12:47.911641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.911591    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:12:47.918885 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:12:47.918850    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6218c303_4ea9_45b7_ad43_17c0b7ee2bff.slice/crio-d74dd52e151653cad93d266945a1ba32aea9bb630d216fc4684dfe43fdae5e65 WatchSource:0}: Error finding container d74dd52e151653cad93d266945a1ba32aea9bb630d216fc4684dfe43fdae5e65: Status 404 returned error can't find the container with id d74dd52e151653cad93d266945a1ba32aea9bb630d216fc4684dfe43fdae5e65
Oct 31 16:12:47.922735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.922690    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.922824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.922765    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:47.922824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:47.922796    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:12:48.182557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.182510    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:48.189047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.189017    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:48.496242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.496144    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:48.501577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.501550    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:48.552396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.552358    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerStarted","Data":"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"}
Oct 31 16:12:48.552586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.552423    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerStarted","Data":"d74dd52e151653cad93d266945a1ba32aea9bb630d216fc4684dfe43fdae5e65"}
Oct 31 16:12:48.553277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553142    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:48.553277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553202    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:48.553277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553227    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:48.553746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553724    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:12:48.553889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553873    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:12:48.553934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.553920    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:12:48.935841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:48.935794    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podStartSLOduration=1.93577959 podStartE2EDuration="1.93577959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:48.935318177 +0000 UTC m=+613.996893363" watchObservedRunningTime="2026-10-31 16:12:48.93577959 +0000 UTC m=+613.997354772"
Oct 31 16:12:49.468896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:49.468856    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:49.474535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:49.474510    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:49.557611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:49.557571    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:49.558267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:49.558249    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:12:59.288936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.288901    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:12:59.292571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.292550    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.370105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.369600    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.370105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.369788    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdpc\" (UniqueName: \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.396568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.396535    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a1685e\""
Oct 31 16:12:59.396764 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.396543    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a1685e-dockercfg-57p6p\""
Oct 31 16:12:59.398814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.398786    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:12:59.446885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.446852    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:12:59.450558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.450537    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.470907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.470879    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.471029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.470948    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdpc\" (UniqueName: \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.471738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.471718    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.503846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.503769    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:12:59.523517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.523482    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdpc\" (UniqueName: \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\") pod \"payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.571704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.571671    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nnq\" (UniqueName: \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.571851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.571730    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.608159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.608132    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:12:59.676638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.673240    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.676638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.673509    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nnq\" (UniqueName: \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.676638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.674691    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.725979 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.725943    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nnq\" (UniqueName: \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\") pod \"payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:12:59.763822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:12:59.763786    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:13:00.009300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.009213    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:13:00.015002 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:00.014977    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa484d9_13c0_4a05_b493_786a96446e90.slice/crio-baa676f8a066a5f10d60f88207c6d0e41c8777dc57b59b2148569f7f9fab8f11 WatchSource:0}: Error finding container baa676f8a066a5f10d60f88207c6d0e41c8777dc57b59b2148569f7f9fab8f11: Status 404 returned error can't find the container with id baa676f8a066a5f10d60f88207c6d0e41c8777dc57b59b2148569f7f9fab8f11
Oct 31 16:13:00.132975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.132944    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:13:00.139802 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:00.139774    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c0100_b01f_4d6f_a68d_40a459879e34.slice/crio-ae8998c53920b7d2a4115f9c2d1b0d7319a251ca59b98af44b65b9df19157b27 WatchSource:0}: Error finding container ae8998c53920b7d2a4115f9c2d1b0d7319a251ca59b98af44b65b9df19157b27: Status 404 returned error can't find the container with id ae8998c53920b7d2a4115f9c2d1b0d7319a251ca59b98af44b65b9df19157b27
Oct 31 16:13:00.635561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.635521    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" event={"ID":"3d4c0100-b01f-4d6f-a68d-40a459879e34","Type":"ContainerStarted","Data":"aaa887874db9c94ba4f8d31a2fc9659d9c4d234b6ee86f2eda3fa1f9b6521bb2"}
Oct 31 16:13:00.636091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.635590    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" event={"ID":"3d4c0100-b01f-4d6f-a68d-40a459879e34","Type":"ContainerStarted","Data":"ae8998c53920b7d2a4115f9c2d1b0d7319a251ca59b98af44b65b9df19157b27"}
Oct 31 16:13:00.636091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.635772    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:13:00.637195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.637167    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" event={"ID":"faa484d9-13c0-4a05-b493-786a96446e90","Type":"ContainerStarted","Data":"78e7e270a9fb0fb4af1831d37602d6b662ffec1703ce565758c8b487edbc3ccd"}
Oct 31 16:13:00.637333 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.637215    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" event={"ID":"faa484d9-13c0-4a05-b493-786a96446e90","Type":"ContainerStarted","Data":"baa676f8a066a5f10d60f88207c6d0e41c8777dc57b59b2148569f7f9fab8f11"}
Oct 31 16:13:00.637333 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.637256    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:13:00.740219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.740158    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" podStartSLOduration=1.7401397109999999 podStartE2EDuration="1.740139711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:00.739687866 +0000 UTC m=+625.801263052" watchObservedRunningTime="2026-10-31 16:13:00.740139711 +0000 UTC m=+625.801714899"
Oct 31 16:13:00.846453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:00.846399    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" podStartSLOduration=1.846384682 podStartE2EDuration="1.846384682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:00.844165861 +0000 UTC m=+625.905741046" watchObservedRunningTime="2026-10-31 16:13:00.846384682 +0000 UTC m=+625.907959867"
Oct 31 16:13:01.646225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.646190    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:13:01.649869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.649846    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.712669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.704407    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f7821a\""
Oct 31 16:13:01.718639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.715017    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:13:01.750317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.750282    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f7821a-dockercfg-tmllb\""
Oct 31 16:13:01.796594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.796559    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:13:01.798884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.798807    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.799125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.799087    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbs7\" (UniqueName: \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.800139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.800121    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:01.850971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.850922    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:13:01.851400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.851348    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\""
Oct 31 16:13:01.851503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.851427    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:13:01.851503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.851479    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:13:01.852396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.852346    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f7821a-serving-cert\""
Oct 31 16:13:01.853919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.853871    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:13:01.900019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.899927    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbs7\" (UniqueName: \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.900173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.900045    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:01.900173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.900137    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8jd\" (UniqueName: \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:01.900314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.900275    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:01.900475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.900358    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:01.900475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.900421    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.901227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.901204    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.953554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.953521    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbs7\" (UniqueName: \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\") pod \"payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:01.963194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:01.963162    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:02.001326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.001284    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.001326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.001326    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8jd\" (UniqueName: \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.001575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.001354    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.001575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.001382    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.001575 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:02.001440    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-serving-cert: secret "maas-api-e2e-worker-w3-f7821a-serving-cert" not found
Oct 31 16:13:02.001575 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:02.001510    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls podName:413a9095-e5d6-4466-a0e1-087b35f7920e nodeName:}" failed. No retries permitted until 2026-10-31 16:13:02.501490581 +0000 UTC m=+627.563065768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls") pod "maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" (UID: "413a9095-e5d6-4466-a0e1-087b35f7920e") : secret "maas-api-e2e-worker-w3-f7821a-serving-cert" not found
Oct 31 16:13:02.002336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.002310    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.004597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.004569    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.054585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.054551    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8jd\" (UniqueName: \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.339271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.339227    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:13:02.346907 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:02.346876    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce917eb_f966_4e0c_aad5_7c5ace3437fa.slice/crio-d66eca21a243e2ec2d51a3f7f10bc110b0e5589c9fdb8d44f0fe3b33c19ac200 WatchSource:0}: Error finding container d66eca21a243e2ec2d51a3f7f10bc110b0e5589c9fdb8d44f0fe3b33c19ac200: Status 404 returned error can't find the container with id d66eca21a243e2ec2d51a3f7f10bc110b0e5589c9fdb8d44f0fe3b33c19ac200
Oct 31 16:13:02.505889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.505849    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.508314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.508292    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.647716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.647556    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" event={"ID":"8ce917eb-f966-4e0c-aad5-7c5ace3437fa","Type":"ContainerStarted","Data":"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"}
Oct 31 16:13:02.647716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.647607    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" event={"ID":"8ce917eb-f966-4e0c-aad5-7c5ace3437fa","Type":"ContainerStarted","Data":"d66eca21a243e2ec2d51a3f7f10bc110b0e5589c9fdb8d44f0fe3b33c19ac200"}
Oct 31 16:13:02.648229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.647718    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:02.713799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.713762    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:02.755069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:02.755001    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" podStartSLOduration=1.754977275 podStartE2EDuration="1.754977275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:02.75283657 +0000 UTC m=+627.814411767" watchObservedRunningTime="2026-10-31 16:13:02.754977275 +0000 UTC m=+627.816552464"
Oct 31 16:13:02.944182 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:02.944102    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:13:03.132870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:03.132839    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:13:03.139722 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:03.139686    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413a9095_e5d6_4466_a0e1_087b35f7920e.slice/crio-101d70165bb7ceb250bc761eff5c2345f36f69126f0029df96b76ca144020f85 WatchSource:0}: Error finding container 101d70165bb7ceb250bc761eff5c2345f36f69126f0029df96b76ca144020f85: Status 404 returned error can't find the container with id 101d70165bb7ceb250bc761eff5c2345f36f69126f0029df96b76ca144020f85
Oct 31 16:13:03.654441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:03.654394    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" event={"ID":"413a9095-e5d6-4466-a0e1-087b35f7920e","Type":"ContainerStarted","Data":"101d70165bb7ceb250bc761eff5c2345f36f69126f0029df96b76ca144020f85"}
Oct 31 16:13:04.662249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:04.662192    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" event={"ID":"413a9095-e5d6-4466-a0e1-087b35f7920e","Type":"ContainerStarted","Data":"f8449833de6d704b40dca395d40be554ea2a0cc397b3fbe9b9bffc6ce661bcc7"}
Oct 31 16:13:04.665249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:04.665155    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:04.773872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:04.773816    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" podStartSLOduration=2.8306212840000002 podStartE2EDuration="3.773799976s" totalImagesPullingTime="943.178692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:04.772078512 +0000 UTC m=+629.833653710" watchObservedRunningTime="2026-10-31 16:13:04.773799976 +0000 UTC m=+629.835375220"
Oct 31 16:13:11.644702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:11.644673    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:13:11.645213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:11.644757    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:13:11.677441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:11.677410    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:13:13.062634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.062593    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:13:13.066343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.066319    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.115363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.115336    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-59357a\""
Oct 31 16:13:13.136612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.136578    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:13:13.140075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.140041    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.140288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.140261    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpclz\" (UniqueName: \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.161165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.161131    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-59357a-dockercfg-w82wp\""
Oct 31 16:13:13.229657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.222042    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:13:13.252640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.250271    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpclz\" (UniqueName: \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.252640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.250368    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.252640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.251441    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.268661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.264011    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.287829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.287799    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:13:13.304276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.304241    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpclz\" (UniqueName: \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\") pod \"payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.385774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.384953    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:13.452779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.452474    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.452779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.452520    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtshq\" (UniqueName: \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.553456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.553418    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.553456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.553459    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtshq\" (UniqueName: \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.554410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.554384    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.610923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.610886    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtshq\" (UniqueName: \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\") pod \"payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:13.657468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.657391    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:13:13.752925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.752890    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:13:13.760143 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:13.760112    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dde006_e3d8_46ed_bb12_9619efbd4414.slice/crio-da9789b580a17b91259fbdb343dddf36d37c142ac344cce2913d398f0cc85aff WatchSource:0}: Error finding container da9789b580a17b91259fbdb343dddf36d37c142ac344cce2913d398f0cc85aff: Status 404 returned error can't find the container with id da9789b580a17b91259fbdb343dddf36d37c142ac344cce2913d398f0cc85aff
Oct 31 16:13:13.889939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:13.889914    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:14.260005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.259921    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:13:14.266135 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:14.266109    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd810ac_b67c_4162_b82f_fc1ccc94ca17.slice/crio-80add3f4491c38db858c56258cfea68228b14d57ed2399c60752a6bb9708b5ce WatchSource:0}: Error finding container 80add3f4491c38db858c56258cfea68228b14d57ed2399c60752a6bb9708b5ce: Status 404 returned error can't find the container with id 80add3f4491c38db858c56258cfea68228b14d57ed2399c60752a6bb9708b5ce
Oct 31 16:13:14.721154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.721114    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" event={"ID":"2dd810ac-b67c-4162-b82f-fc1ccc94ca17","Type":"ContainerStarted","Data":"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"}
Oct 31 16:13:14.721346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.721200    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" event={"ID":"2dd810ac-b67c-4162-b82f-fc1ccc94ca17","Type":"ContainerStarted","Data":"80add3f4491c38db858c56258cfea68228b14d57ed2399c60752a6bb9708b5ce"}
Oct 31 16:13:14.721346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.721263    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:14.723060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.723019    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" event={"ID":"20dde006-e3d8-46ed-bb12-9619efbd4414","Type":"ContainerStarted","Data":"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"}
Oct 31 16:13:14.723183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.723099    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" event={"ID":"20dde006-e3d8-46ed-bb12-9619efbd4414","Type":"ContainerStarted","Data":"da9789b580a17b91259fbdb343dddf36d37c142ac344cce2913d398f0cc85aff"}
Oct 31 16:13:14.723246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.723202    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:14.838027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:14.837559    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" podStartSLOduration=1.8375378850000001 podStartE2EDuration="1.837537885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:14.833666715 +0000 UTC m=+639.895241903" watchObservedRunningTime="2026-10-31 16:13:14.837537885 +0000 UTC m=+639.899113074"
Oct 31 16:13:15.417916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.417858    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" podStartSLOduration=2.4178411349999998 podStartE2EDuration="2.417841135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:14.943513053 +0000 UTC m=+640.005088230" watchObservedRunningTime="2026-10-31 16:13:15.417841135 +0000 UTC m=+640.479416321"
Oct 31 16:13:15.418296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.417985    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:13:15.421741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.421713    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.477552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.477522    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:13:15.487577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.487539    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx2v\" (UniqueName: \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.487915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.487894    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.518920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.518890    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ef6d93-dockercfg-2mptm\""
Oct 31 16:13:15.518920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.518890    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ef6d93\""
Oct 31 16:13:15.590639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.590466    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx2v\" (UniqueName: \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.590845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.590650    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.595668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.591601    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.669720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.667579    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx2v\" (UniqueName: \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\") pod \"payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:15.764864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:15.764824    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:16.150640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:16.150375    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:13:16.744648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:16.743687    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" event={"ID":"4727544f-ec71-4aa1-96cd-289815522f8e","Type":"ContainerStarted","Data":"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"}
Oct 31 16:13:16.744648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:16.743762    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" event={"ID":"4727544f-ec71-4aa1-96cd-289815522f8e","Type":"ContainerStarted","Data":"834b8e4c0cf0ce03a3c53a174effe460180f1e3f15cbaa4fdd6daea329645295"}
Oct 31 16:13:16.745466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:16.745398    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:16.859458 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:16.857011    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" podStartSLOduration=1.8569898409999999 podStartE2EDuration="1.856989841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:16.856042233 +0000 UTC m=+641.917617439" watchObservedRunningTime="2026-10-31 16:13:16.856989841 +0000 UTC m=+641.918565028"
Oct 31 16:13:23.600091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.596466    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:13:23.602738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.602703    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.664788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.664751    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:13:23.698476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.698433    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-4fe3a4-serving-cert\""
Oct 31 16:13:23.706063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.706013    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.706248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.706188    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.706339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.706317    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dzx\" (UniqueName: \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.706412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.706387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.807147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.807114    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dzx\" (UniqueName: \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.807147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.807155    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.807436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.807207    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.807436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.807267    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.807436 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:23.807347    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-serving-cert: secret "maas-api-e2e-worker-w1-4fe3a4-serving-cert" not found
Oct 31 16:13:23.807436 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:23.807421    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls podName:51bba813-c18f-4d27-b39f-b40fad3e81e8 nodeName:}" failed. No retries permitted until 2026-10-31 16:13:24.307399868 +0000 UTC m=+649.368975053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls") pod "maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" (UID: "51bba813-c18f-4d27-b39f-b40fad3e81e8") : secret "maas-api-e2e-worker-w1-4fe3a4-serving-cert" not found
Oct 31 16:13:23.808208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.808175    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.809818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.809798    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:23.879331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:23.879198    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dzx\" (UniqueName: \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:24.313763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.313733    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:24.315902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.315878    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:24.517612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.517516    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:24.811610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.811578    4417 generic.go:356] "Generic (PLEG): container finished" podID="80599a52-229a-4d25-a347-687580a623e8" containerID="62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a" exitCode=0
Oct 31 16:13:24.812087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.811654    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerDied","Data":"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"}
Oct 31 16:13:24.812087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.812046    4417 scope.go:122] "RemoveContainer" containerID="62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"
Oct 31 16:13:24.813384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.813352    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:24.813498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.813439    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:24.813498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.813487    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:24.915146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:24.915118    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:13:24.922893 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:24.922851    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bba813_c18f_4d27_b39f_b40fad3e81e8.slice/crio-e29bb29194d91f80920662640a798879ee2b05de4f9d44a3506e8d553eedb0ae WatchSource:0}: Error finding container e29bb29194d91f80920662640a798879ee2b05de4f9d44a3506e8d553eedb0ae: Status 404 returned error can't find the container with id e29bb29194d91f80920662640a798879ee2b05de4f9d44a3506e8d553eedb0ae
Oct 31 16:13:25.731346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.731260    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:13:25.732523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.732498    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:13:25.818202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.818174    4417 generic.go:356] "Generic (PLEG): container finished" podID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerID="32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb" exitCode=0
Oct 31 16:13:25.818650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.818248    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerDied","Data":"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"}
Oct 31 16:13:25.819084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.819063    4417 scope.go:122] "RemoveContainer" containerID="32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"
Oct 31 16:13:25.820697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.820667    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:25.820751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.820739    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:25.820785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.820774    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:25.822240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.821174    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerStarted","Data":"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"}
Oct 31 16:13:25.824988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:25.824955    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" event={"ID":"51bba813-c18f-4d27-b39f-b40fad3e81e8","Type":"ContainerStarted","Data":"e29bb29194d91f80920662640a798879ee2b05de4f9d44a3506e8d553eedb0ae"}
Oct 31 16:13:26.189360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.189268    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:13:26.189777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.189685    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 16:13:26.189777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.189729    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 16:13:26.831261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.831223    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" event={"ID":"51bba813-c18f-4d27-b39f-b40fad3e81e8","Type":"ContainerStarted","Data":"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"}
Oct 31 16:13:26.831862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.831843    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:26.833252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.833226    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerStarted","Data":"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"}
Oct 31 16:13:26.967641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:26.953405    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" podStartSLOduration=2.968892791 podStartE2EDuration="3.95338364s" totalImagesPullingTime="984.490849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:26.951093106 +0000 UTC m=+652.012668293" watchObservedRunningTime="2026-10-31 16:13:26.95338364 +0000 UTC m=+652.014958829"
Oct 31 16:13:27.186182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:27.186078    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 16:13:27.186460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:27.186423    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 16:13:27.469430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:27.469339    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:13:27.469812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:27.469790    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 16:13:27.469963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:27.469940    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 16:13:28.185037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:28.184998    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 16:13:28.185681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:28.185648    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 16:13:28.469107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:28.469012    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 16:13:28.469372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:28.469339    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 16:13:28.827643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:28.823307    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:13:29.184654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:29.184082    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 16:13:29.184654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:29.184150    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 16:13:29.484723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:29.475011    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 16:13:29.484723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:29.475076    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 16:13:30.184120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:30.184070    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 16:13:30.184346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:30.184163    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 16:13:30.473706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:30.473087    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 16:13:30.473706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:30.473167    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 16:13:31.256052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.256008    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:13:31.258323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.258302    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:13:31.272389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.272258    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:13:31.468869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.468823    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 16:13:31.469354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.468928    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 16:13:31.672352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.667267    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:13:31.704638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.701421    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.776169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.776137    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:13:31.826113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826078    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826131    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826262    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826322    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826339    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz2h\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826369    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826458    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826497    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.826605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.826524    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.927756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927668    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.927756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927717    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927762    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927814    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927832    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz2h\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927874    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927911    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927956    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.927999    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.928153    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.928258    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.928456    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.928512    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.928715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.928680    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.930314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.930288    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.930656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.930637    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.991601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.991528    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:31.994853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:31.994773    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz2h\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:32.022372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.021884    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:32.475966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.475938    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:13:32.478759 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:32.478721    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef7b5ed_7f7a_48d2_b253_a5e06e0272ca.slice/crio-e3d1467d24118f68777ffe4e23d92ff7cefa008acbda4514ae995255dff9a3a7 WatchSource:0}: Error finding container e3d1467d24118f68777ffe4e23d92ff7cefa008acbda4514ae995255dff9a3a7: Status 404 returned error can't find the container with id e3d1467d24118f68777ffe4e23d92ff7cefa008acbda4514ae995255dff9a3a7
Oct 31 16:13:32.481379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.481342    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:32.481502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.481424    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:32.481502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.481467    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:32.564257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.564217    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:13:32.565739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.565502    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:13:32.570181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:32.570158    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:13:32.968466 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:32.968409    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:13:33.015994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.015952    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" event={"ID":"def7b5ed-7f7a-48d2-b253-a5e06e0272ca","Type":"ContainerStarted","Data":"12b0f14580a0bcf2d2b541c6dc3b1aea4fcc7f6674d8e5dc1c2271c0fdaf9e92"}
Oct 31 16:13:33.016146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.016012    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" event={"ID":"def7b5ed-7f7a-48d2-b253-a5e06e0272ca","Type":"ContainerStarted","Data":"e3d1467d24118f68777ffe4e23d92ff7cefa008acbda4514ae995255dff9a3a7"}
Oct 31 16:13:33.022682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.022658    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:33.024206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.024181    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:33.024313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.024227    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:33.130631 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.130559    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podStartSLOduration=2.130545428 podStartE2EDuration="2.130545428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:33.130386266 +0000 UTC m=+658.191961458" watchObservedRunningTime="2026-10-31 16:13:33.130545428 +0000 UTC m=+658.192120613"
Oct 31 16:13:33.602389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.602353    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:13:33.606232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.606207    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.671645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.667954    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-59357a-serving-cert\""
Oct 31 16:13:33.672007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.671983    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:13:33.790642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.790243    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2wf\" (UniqueName: \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.790642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.790339    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.790642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.790390    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.790642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.790478    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.891326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.891233    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2wf\" (UniqueName: \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.892661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.891595    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.892926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.892902    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.893147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.893128    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.894151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.894128    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.895078 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:33.895060    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-serving-cert: secret "maas-api-e2e-worker-w4-59357a-serving-cert" not found
Oct 31 16:13:33.900950 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:13:33.900932    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls podName:d5e05893-d2c1-48b6-ad63-1abc3eb72e3f nodeName:}" failed. No retries permitted until 2026-10-31 16:13:34.400903874 +0000 UTC m=+659.462479052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls") pod "maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" (UID: "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f") : secret "maas-api-e2e-worker-w4-59357a-serving-cert" not found
Oct 31 16:13:33.922145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.922110    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:33.973389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:33.973350    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2wf\" (UniqueName: \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:34.006177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.006129    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:13:34.023015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.022979    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:34.025758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.023041    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:34.409432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.409394    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:34.412706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.412676    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:34.539650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.533116    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:34.607824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.607765    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:34.608243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:34.607827    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:35.022724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.022695    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:35.022922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.022742    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:35.030426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.030399    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:13:35.030606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.030583    4417 generic.go:356] "Generic (PLEG): container finished" podID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerID="6c0496446a5747a0565b44bce25c76af942f839c297f5949df9a20880c023923" exitCode=0
Oct 31 16:13:35.030939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.030917    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerDied","Data":"6c0496446a5747a0565b44bce25c76af942f839c297f5949df9a20880c023923"}
Oct 31 16:13:35.031205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.031189    4417 scope.go:122] "RemoveContainer" containerID="6c0496446a5747a0565b44bce25c76af942f839c297f5949df9a20880c023923"
Oct 31 16:13:35.032523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.032492    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:35.032607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.032558    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:35.032607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:35.032595    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:35.033406 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:13:35.033384    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e05893_d2c1_48b6_ad63_1abc3eb72e3f.slice/crio-3e012077801a60655b4cd744615bcd79abd0605c413195b6e9e0eb0f7f1fcd9e WatchSource:0}: Error finding container 3e012077801a60655b4cd744615bcd79abd0605c413195b6e9e0eb0f7f1fcd9e: Status 404 returned error can't find the container with id 3e012077801a60655b4cd744615bcd79abd0605c413195b6e9e0eb0f7f1fcd9e
Oct 31 16:13:36.022902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.022865    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:36.023394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.023356    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:36.076821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.075593    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" event={"ID":"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f","Type":"ContainerStarted","Data":"3e012077801a60655b4cd744615bcd79abd0605c413195b6e9e0eb0f7f1fcd9e"}
Oct 31 16:13:36.092828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.090255    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerStarted","Data":"ae6b7a60d2391780d3f8738654ae7bd194e362c46019cd634000465a2f18f138"}
Oct 31 16:13:36.500641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.500075    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:13:36.500641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.500208    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:13:36.500641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:36.500251    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 16:13:37.022898 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.022866    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:37.023135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.023114    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:37.126932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.126892    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" event={"ID":"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f","Type":"ContainerStarted","Data":"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"}
Oct 31 16:13:37.127107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.126957    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:37.257382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.257314    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" podStartSLOduration=3.274269685 podStartE2EDuration="4.257293603s" totalImagesPullingTime="983.023918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:37.254089949 +0000 UTC m=+662.315665136" watchObservedRunningTime="2026-10-31 16:13:37.257293603 +0000 UTC m=+662.318868790"
Oct 31 16:13:37.499440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.499403    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:13:37.499737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:37.499707    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 16:13:38.023168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:38.023121    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 16:13:38.023832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:38.023780    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 16:13:38.503639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:38.503057    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:13:38.503639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:38.503121    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 16:13:39.026574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:39.026542    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:39.027092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:39.027072    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:39.027812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:39.027786    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:13:39.497087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:39.497048    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:13:39.497285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:39.497126    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 16:13:40.496417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:40.496383    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 16:13:40.496835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:40.496434    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 16:13:41.588293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:41.588259    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:13:41.589132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:41.589104    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:13:41.590386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:41.590363    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:13:43.175022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.174933    4417 generic.go:356] "Generic (PLEG): container finished" podID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerID="e2c8dfb191f63b0e66c666dbcb4562a55efe3acb0dda8f755f85a038fb836c23" exitCode=0
Oct 31 16:13:43.175022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.175007    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerDied","Data":"e2c8dfb191f63b0e66c666dbcb4562a55efe3acb0dda8f755f85a038fb836c23"}
Oct 31 16:13:43.175606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.175590    4417 scope.go:122] "RemoveContainer" containerID="e2c8dfb191f63b0e66c666dbcb4562a55efe3acb0dda8f755f85a038fb836c23"
Oct 31 16:13:43.176868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.176841    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:43.176947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.176911    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:43.176985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:43.176943    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:13:44.159186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:44.159153    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:13:44.188225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:44.188192    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerStarted","Data":"001eb2c700e867dac46f77d9ff90902fc47b9b27eeddb1751f195254e17510ae"}
Oct 31 16:13:44.565056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:44.565015    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:13:44.566391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:44.566363    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 16:13:44.566540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:44.566419    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 16:13:45.565711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:45.565665    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 16:13:45.566118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:45.565751    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 16:13:46.565088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:46.565046    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 16:13:46.565286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:46.565116    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 16:13:47.565355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:47.565318    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 16:13:47.565969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:47.565395    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 16:13:48.552764 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:48.552721    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:13:48.565540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:48.565505    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 16:13:48.565990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:48.565562    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 16:13:49.569772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:49.569740    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:13:49.570656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:13:49.570638    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:14:02.944202 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:14:02.944165    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:14:32.944127 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:14:32.944083    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:15:00.166626 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.166579    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:00.170454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.170429    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.182000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.181976    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:00.185452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.185434    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:00.185565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.185547    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.188807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.188791    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.225816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.225793    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:15:00.228305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.228284    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.228411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.228314    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8w5v\" (UniqueName: \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.228461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.228413    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.228504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.228477    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slppd\" (UniqueName: \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.262046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.262013    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:00.309840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.309777    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:00.313094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.313059    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:00.330252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330220    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.330716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330296    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.330716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330357    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slppd\" (UniqueName: \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.330716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330486    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxmq\" (UniqueName: \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.330716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330658    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.330716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330693    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8w5v\" (UniqueName: \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.330980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.330961    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.331326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.331309    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.389737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.389694    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8w5v\" (UniqueName: \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.389997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.389972    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slppd\" (UniqueName: \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.432054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.431972    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.432218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.432080    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxmq\" (UniqueName: \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.432683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.432658    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.483203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.483156    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:00.485252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.485225    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxmq\" (UniqueName: \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\") pod \"maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.499658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.499633    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:00.505936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.505912    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:00.918362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.918322    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:00.949568 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:15:00.949430    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36be6acb_a4b4_4f53_a32e_b02f44ec3d0a.slice/crio-7cc2e7ce08cd6f45481583b73f4dba021860f55420bc4aa98b76e93f356c831f WatchSource:0}: Error finding container 7cc2e7ce08cd6f45481583b73f4dba021860f55420bc4aa98b76e93f356c831f: Status 404 returned error can't find the container with id 7cc2e7ce08cd6f45481583b73f4dba021860f55420bc4aa98b76e93f356c831f
Oct 31 16:15:00.951863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.951830    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:15:00.966958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.966776    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:00.969540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:00.969519    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:00.969733 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:15:00.969699    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d5a945_17c2_4725_96dc_b1a049175adf.slice/crio-2f594b5464b33df83a9abd2fff7060fd10fd8263d1db447a23f601abc8c26caa WatchSource:0}: Error finding container 2f594b5464b33df83a9abd2fff7060fd10fd8263d1db447a23f601abc8c26caa: Status 404 returned error can't find the container with id 2f594b5464b33df83a9abd2fff7060fd10fd8263d1db447a23f601abc8c26caa
Oct 31 16:15:00.972524 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:15:00.972503    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e88015_631a_4f62_b14c_42e8fc2ab672.slice/crio-0a680484e4b99d6361833b0cd0c24d0aecf856cc6724198d58739eda1064d674 WatchSource:0}: Error finding container 0a680484e4b99d6361833b0cd0c24d0aecf856cc6724198d58739eda1064d674: Status 404 returned error can't find the container with id 0a680484e4b99d6361833b0cd0c24d0aecf856cc6724198d58739eda1064d674
Oct 31 16:15:01.596949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:01.596913    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerStarted","Data":"7cc2e7ce08cd6f45481583b73f4dba021860f55420bc4aa98b76e93f356c831f"}
Oct 31 16:15:01.598147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:01.598124    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerStarted","Data":"0a680484e4b99d6361833b0cd0c24d0aecf856cc6724198d58739eda1064d674"}
Oct 31 16:15:01.599273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:01.599251    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerStarted","Data":"2f594b5464b33df83a9abd2fff7060fd10fd8263d1db447a23f601abc8c26caa"}
Oct 31 16:15:02.943996 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:02.943957    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:15:06.089252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.089216    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:15:06.092891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.092868    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132405    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132461    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132654    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132771    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk84t\" (UniqueName: \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132868    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.132926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.132900    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.143114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.143078    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"openshift-service-ca.crt\""
Oct 31 16:15:06.149718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.147146    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:15:06.189349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.189307    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"default-dockercfg-zbwr8\""
Oct 31 16:15:06.189880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.189863    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"simulator-w4-59357a-kserve-self-signed-certs\""
Oct 31 16:15:06.252903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.250158    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"kube-root-ca.crt\""
Oct 31 16:15:06.259933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.259890    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.260118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260000    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk84t\" (UniqueName: \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.260118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260064    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.260224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260146    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.260282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260271    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.260334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260308    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.261180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.260844    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.270836 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.263954    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.270836 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.264516    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.275481 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.275415    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.278235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.278206    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.316292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.316264    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk84t\" (UniqueName: \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\") pod \"simulator-w4-59357a-kserve-849d764dc7-4f2p6\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") " pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.404464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.404424    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:06.630115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.630074    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerStarted","Data":"3001f9b588b0978401d15d7cd933019add3dd06389e44d0c3142310f1f45eb4d"}
Oct 31 16:15:06.631540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.631510    4417 generic.go:356] "Generic (PLEG): container finished" podID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerID="264bfd608ff9bdd251344e8d1701534915558af8f47cdde134da1106386a9cde" exitCode=60
Oct 31 16:15:06.631697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.631600    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerDied","Data":"264bfd608ff9bdd251344e8d1701534915558af8f47cdde134da1106386a9cde"}
Oct 31 16:15:06.631847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.631829    4417 scope.go:122] "RemoveContainer" containerID="264bfd608ff9bdd251344e8d1701534915558af8f47cdde134da1106386a9cde"
Oct 31 16:15:06.633147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.633127    4417 generic.go:356] "Generic (PLEG): container finished" podID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerID="c237314cb4e49f999a5b30bd4507489b567426c67c44efd9a3f59c9223156f3d" exitCode=60
Oct 31 16:15:06.633247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.633208    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerDied","Data":"c237314cb4e49f999a5b30bd4507489b567426c67c44efd9a3f59c9223156f3d"}
Oct 31 16:15:06.633399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.633383    4417 scope.go:122] "RemoveContainer" containerID="c237314cb4e49f999a5b30bd4507489b567426c67c44efd9a3f59c9223156f3d"
Oct 31 16:15:06.736631 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.736557    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" podStartSLOduration=1.916652677 podStartE2EDuration="6.736535842s" totalImagesPullingTime="4.819883165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:06.733174263 +0000 UTC m=+751.794749448" watchObservedRunningTime="2026-10-31 16:15:06.736535842 +0000 UTC m=+751.798111029"
Oct 31 16:15:06.782795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:06.782761    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:15:06.836510 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:15:06.836462    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfd98130_d605_4f94_8ba5_eb8afa6a8390.slice/crio-3ad527d473d0f07c7f8c9c10b7709a21adf411ce8bf7a6bdacb2a560526860d7 WatchSource:0}: Error finding container 3ad527d473d0f07c7f8c9c10b7709a21adf411ce8bf7a6bdacb2a560526860d7: Status 404 returned error can't find the container with id 3ad527d473d0f07c7f8c9c10b7709a21adf411ce8bf7a6bdacb2a560526860d7
Oct 31 16:15:07.646306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.646271    4417 generic.go:356] "Generic (PLEG): container finished" podID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerID="3001f9b588b0978401d15d7cd933019add3dd06389e44d0c3142310f1f45eb4d" exitCode=60
Oct 31 16:15:07.646717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.646362    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerDied","Data":"3001f9b588b0978401d15d7cd933019add3dd06389e44d0c3142310f1f45eb4d"}
Oct 31 16:15:07.646787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.646723    4417 scope.go:122] "RemoveContainer" containerID="3001f9b588b0978401d15d7cd933019add3dd06389e44d0c3142310f1f45eb4d"
Oct 31 16:15:07.648891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.648852    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerStarted","Data":"3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"}
Oct 31 16:15:07.649159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.649141    4417 scope.go:122] "RemoveContainer" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"
Oct 31 16:15:07.649406 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:07.649387    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz_odh-ai-gateway-infra(a1d5a945-17c2-4725-96dc-b1a049175adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf"
Oct 31 16:15:07.650907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.650880    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" event={"ID":"cfd98130-d605-4f94-8ba5-eb8afa6a8390","Type":"ContainerStarted","Data":"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"}
Oct 31 16:15:07.651014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.650921    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" event={"ID":"cfd98130-d605-4f94-8ba5-eb8afa6a8390","Type":"ContainerStarted","Data":"3ad527d473d0f07c7f8c9c10b7709a21adf411ce8bf7a6bdacb2a560526860d7"}
Oct 31 16:15:07.653205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.653185    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerStarted","Data":"1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"}
Oct 31 16:15:07.653506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.653488    4417 scope.go:122] "RemoveContainer" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"
Oct 31 16:15:07.653790 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:07.653764    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm_odh-ai-gateway-infra(36be6acb-a4b4-4f53-a32e-b02f44ec3d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"
Oct 31 16:15:07.744663 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:07.744600    4417 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36be6acb_a4b4_4f53_a32e_b02f44ec3d0a.slice/crio-conmon-1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:15:07.856026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:07.855967    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podStartSLOduration=1.8559525300000002 podStartE2EDuration="1.85595253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:07.854306571 +0000 UTC m=+752.915881756" watchObservedRunningTime="2026-10-31 16:15:07.85595253 +0000 UTC m=+752.917527760"
Oct 31 16:15:08.405219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.405183    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:08.662606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.662514    4417 generic.go:356] "Generic (PLEG): container finished" podID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerID="66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df" exitCode=60
Oct 31 16:15:08.663108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.662604    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerDied","Data":"66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df"}
Oct 31 16:15:08.663108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.662699    4417 scope.go:122] "RemoveContainer" containerID="3001f9b588b0978401d15d7cd933019add3dd06389e44d0c3142310f1f45eb4d"
Oct 31 16:15:08.663108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.662968    4417 scope.go:122] "RemoveContainer" containerID="66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df"
Oct 31 16:15:08.663331 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:08.663305    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss_odh-ai-gateway-infra(38e88015-631a-4f62-b14c-42e8fc2ab672)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672"
Oct 31 16:15:08.664773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.664748    4417 generic.go:356] "Generic (PLEG): container finished" podID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7" exitCode=60
Oct 31 16:15:08.664874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.664833    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerDied","Data":"3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"}
Oct 31 16:15:08.665123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.665107    4417 scope.go:122] "RemoveContainer" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"
Oct 31 16:15:08.665516 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:08.665480    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz_odh-ai-gateway-infra(a1d5a945-17c2-4725-96dc-b1a049175adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf"
Oct 31 16:15:08.667257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.667236    4417 generic.go:356] "Generic (PLEG): container finished" podID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b" exitCode=60
Oct 31 16:15:08.667363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.667339    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerDied","Data":"1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"}
Oct 31 16:15:08.667605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.667586    4417 scope.go:122] "RemoveContainer" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"
Oct 31 16:15:08.667902 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:08.667879    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm_odh-ai-gateway-infra(36be6acb-a4b4-4f53-a32e-b02f44ec3d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"
Oct 31 16:15:08.676148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.676125    4417 scope.go:122] "RemoveContainer" containerID="264bfd608ff9bdd251344e8d1701534915558af8f47cdde134da1106386a9cde"
Oct 31 16:15:08.688317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:08.688289    4417 scope.go:122] "RemoveContainer" containerID="c237314cb4e49f999a5b30bd4507489b567426c67c44efd9a3f59c9223156f3d"
Oct 31 16:15:09.675136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:09.675107    4417 scope.go:122] "RemoveContainer" containerID="66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df"
Oct 31 16:15:09.675604 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:09.675445    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss_odh-ai-gateway-infra(38e88015-631a-4f62-b14c-42e8fc2ab672)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672"
Oct 31 16:15:09.676862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:09.676836    4417 scope.go:122] "RemoveContainer" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"
Oct 31 16:15:09.677143 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:09.677122    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz_odh-ai-gateway-infra(a1d5a945-17c2-4725-96dc-b1a049175adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf"
Oct 31 16:15:09.678313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:09.678295    4417 scope.go:122] "RemoveContainer" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"
Oct 31 16:15:09.678542 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:09.678525    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm_odh-ai-gateway-infra(36be6acb-a4b4-4f53-a32e-b02f44ec3d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"
Oct 31 16:15:16.405273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:16.405236    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:16.418274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:16.418245    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:16.728067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:16.727981    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:15:18.042220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:18.042190    4417 scope.go:122] "RemoveContainer" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"
Oct 31 16:15:18.729970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:18.729888    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerStarted","Data":"b1da2d050de3a9302fd9d90a279d6cf5569102906b3405acfa4d35c4a5c27651"}
Oct 31 16:15:18.834197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:18.834134    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" podStartSLOduration=14.415515427 podStartE2EDuration="18.83411645s" totalImagesPullingTime="4.418601023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:18.833206997 +0000 UTC m=+763.894782182" watchObservedRunningTime="2026-10-31 16:15:18.83411645 +0000 UTC m=+763.895691660"
Oct 31 16:15:19.737812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:19.737777    4417 generic.go:356] "Generic (PLEG): container finished" podID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerID="b1da2d050de3a9302fd9d90a279d6cf5569102906b3405acfa4d35c4a5c27651" exitCode=60
Oct 31 16:15:19.738298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:19.737860    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerDied","Data":"b1da2d050de3a9302fd9d90a279d6cf5569102906b3405acfa4d35c4a5c27651"}
Oct 31 16:15:19.738298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:19.737942    4417 scope.go:122] "RemoveContainer" containerID="3e8980709704388de55c236adbd7771a8fdeac64ca075dc6ee0fc72868fe7de7"
Oct 31 16:15:19.738407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:19.738312    4417 scope.go:122] "RemoveContainer" containerID="b1da2d050de3a9302fd9d90a279d6cf5569102906b3405acfa4d35c4a5c27651"
Oct 31 16:15:19.738675 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:19.738642    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz_odh-ai-gateway-infra(a1d5a945-17c2-4725-96dc-b1a049175adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf"
Oct 31 16:15:19.842601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:19.842566    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:20.879314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.879291    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:20.926719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.926675    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\") pod \"a1d5a945-17c2-4725-96dc-b1a049175adf\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") "
Oct 31 16:15:20.926899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.926788    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\" (UniqueName: \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\") pod \"a1d5a945-17c2-4725-96dc-b1a049175adf\" (UID: \"a1d5a945-17c2-4725-96dc-b1a049175adf\") "
Oct 31 16:15:20.927071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.927046    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca" pod "a1d5a945-17c2-4725-96dc-b1a049175adf" (UID: "a1d5a945-17c2-4725-96dc-b1a049175adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:20.927223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.927208    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1d5a945-17c2-4725-96dc-b1a049175adf-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:20.928886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:20.928865    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq" pod "a1d5a945-17c2-4725-96dc-b1a049175adf" (UID: "a1d5a945-17c2-4725-96dc-b1a049175adf"). InnerVolumeSpecName "kube-api-access-gbxmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:21.028447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.028359    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxmq\" (UniqueName: \"kubernetes.io/projected/a1d5a945-17c2-4725-96dc-b1a049175adf-kube-api-access-gbxmq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:21.042031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.041999    4417 scope.go:122] "RemoveContainer" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"
Oct 31 16:15:21.752589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.752545    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerStarted","Data":"b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f"}
Oct 31 16:15:21.753054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.753033    4417 scope.go:122] "RemoveContainer" containerID="b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f"
Oct 31 16:15:21.753323 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:21.753301    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm_odh-ai-gateway-infra(36be6acb-a4b4-4f53-a32e-b02f44ec3d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"
Oct 31 16:15:21.753992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.753971    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz" event={"ID":"a1d5a945-17c2-4725-96dc-b1a049175adf","Type":"ContainerDied","Data":"2f594b5464b33df83a9abd2fff7060fd10fd8263d1db447a23f601abc8c26caa"}
Oct 31 16:15:21.754071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.753997    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"
Oct 31 16:15:21.754071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.754018    4417 scope.go:122] "RemoveContainer" containerID="b1da2d050de3a9302fd9d90a279d6cf5569102906b3405acfa4d35c4a5c27651"
Oct 31 16:15:21.958732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.958698    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:21.961024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:21.960991    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a1685e-29891055-tvtcz"]
Oct 31 16:15:22.042230 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.042199    4417 scope.go:122] "RemoveContainer" containerID="66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df"
Oct 31 16:15:22.762123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.762031    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerStarted","Data":"d2da26c615fdf77773408dd7a8afcd7a0112c35c9ee7ab7111c0bf122d79168a"}
Oct 31 16:15:22.765090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.765064    4417 generic.go:356] "Generic (PLEG): container finished" podID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerID="b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f" exitCode=60
Oct 31 16:15:22.765271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.765138    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerDied","Data":"b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f"}
Oct 31 16:15:22.765271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.765201    4417 scope.go:122] "RemoveContainer" containerID="1bda7c314e504e8257605123b7d1527fc4e4b924392d322e0679893723f0044b"
Oct 31 16:15:22.765575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.765558    4417 scope.go:122] "RemoveContainer" containerID="b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f"
Oct 31 16:15:22.765879 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:22.765857    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm_odh-ai-gateway-infra(36be6acb-a4b4-4f53-a32e-b02f44ec3d0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"
Oct 31 16:15:22.868686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:22.868650    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:23.049173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.049129    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" path="/var/lib/kubelet/pods/a1d5a945-17c2-4725-96dc-b1a049175adf/volumes"
Oct 31 16:15:23.775948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.775911    4417 generic.go:356] "Generic (PLEG): container finished" podID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerID="d2da26c615fdf77773408dd7a8afcd7a0112c35c9ee7ab7111c0bf122d79168a" exitCode=60
Oct 31 16:15:23.776073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.775951    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerDied","Data":"d2da26c615fdf77773408dd7a8afcd7a0112c35c9ee7ab7111c0bf122d79168a"}
Oct 31 16:15:23.776073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.776028    4417 scope.go:122] "RemoveContainer" containerID="66cac48fc6c1eb1de0d908224d0e5dab5a1e27906ba9bb6110196e7d67bba9df"
Oct 31 16:15:23.776432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.776410    4417 scope.go:122] "RemoveContainer" containerID="d2da26c615fdf77773408dd7a8afcd7a0112c35c9ee7ab7111c0bf122d79168a"
Oct 31 16:15:23.776748 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:23.776723    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss_odh-ai-gateway-infra(38e88015-631a-4f62-b14c-42e8fc2ab672)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672"
Oct 31 16:15:23.877413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.877316    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:23.935525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:23.935497    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:24.060035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.059351    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\") pod \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") "
Oct 31 16:15:24.060035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.059437    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\" (UniqueName: \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\") pod \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\" (UID: \"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a\") "
Oct 31 16:15:24.060035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.059752    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca" pod "36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" (UID: "36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:24.060035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.059926    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.061965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.061930    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v" pod "36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" (UID: "36be6acb-a4b4-4f53-a32e-b02f44ec3d0a"). InnerVolumeSpecName "kube-api-access-p8w5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:24.160978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.160943    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8w5v\" (UniqueName: \"kubernetes.io/projected/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a-kube-api-access-p8w5v\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.785845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.785817    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"
Oct 31 16:15:24.785951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.785841    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm" event={"ID":"36be6acb-a4b4-4f53-a32e-b02f44ec3d0a","Type":"ContainerDied","Data":"7cc2e7ce08cd6f45481583b73f4dba021860f55420bc4aa98b76e93f356c831f"}
Oct 31 16:15:24.785951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.785901    4417 scope.go:122] "RemoveContainer" containerID="b8acec5309d428e7c0d1fc6160e65ffdfe742e684165d882ddeed176b2a30d3f"
Oct 31 16:15:24.936704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.936587    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:24.990884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.990850    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:24.996827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:24.996799    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-59357a-29891055-pvwbm"]
Oct 31 16:15:25.048925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.048897    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" path="/var/lib/kubelet/pods/36be6acb-a4b4-4f53-a32e-b02f44ec3d0a/volumes"
Oct 31 16:15:25.072031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.072002    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\") pod \"38e88015-631a-4f62-b14c-42e8fc2ab672\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") "
Oct 31 16:15:25.072418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.072050    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\" (UniqueName: \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\") pod \"38e88015-631a-4f62-b14c-42e8fc2ab672\" (UID: \"38e88015-631a-4f62-b14c-42e8fc2ab672\") "
Oct 31 16:15:25.072461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.072421    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca" pod "38e88015-631a-4f62-b14c-42e8fc2ab672" (UID: "38e88015-631a-4f62-b14c-42e8fc2ab672"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:25.074248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.074224    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd" pod "38e88015-631a-4f62-b14c-42e8fc2ab672" (UID: "38e88015-631a-4f62-b14c-42e8fc2ab672"). InnerVolumeSpecName "kube-api-access-slppd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:25.173019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.172974    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e88015-631a-4f62-b14c-42e8fc2ab672-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.173241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.173083    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slppd\" (UniqueName: \"kubernetes.io/projected/38e88015-631a-4f62-b14c-42e8fc2ab672-kube-api-access-slppd\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.793924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.793889    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss" event={"ID":"38e88015-631a-4f62-b14c-42e8fc2ab672","Type":"ContainerDied","Data":"0a680484e4b99d6361833b0cd0c24d0aecf856cc6724198d58739eda1064d674"}
Oct 31 16:15:25.794122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.793949    4417 scope.go:122] "RemoveContainer" containerID="d2da26c615fdf77773408dd7a8afcd7a0112c35c9ee7ab7111c0bf122d79168a"
Oct 31 16:15:25.794122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.793907    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"
Oct 31 16:15:25.956339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.956299    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:25.961212 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:25.961185    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891055-dnqss"]
Oct 31 16:15:27.048714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:15:27.048676    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" path="/var/lib/kubelet/pods/38e88015-631a-4f62-b14c-42e8fc2ab672/volumes"
Oct 31 16:15:32.944559 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:15:32.944512    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:16:02.126132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126094    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:16:02.126567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126544    4417 cpu_manager.go:414] "containerMap: removing container" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.126567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126555    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126572    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126577    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126584    4417 cpu_manager.go:414] "containerMap: removing container" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126589    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126598    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126603    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126627    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126632    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126640    4417 cpu_manager.go:414] "containerMap: removing container" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126645    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126651    4417 cpu_manager.go:414] "containerMap: removing container" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126655    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126664    4417 cpu_manager.go:414] "containerMap: removing container" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126668    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126677    4417 cpu_manager.go:414] "containerMap: removing container" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.126689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126681    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126767    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126775    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126780    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126786    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126792    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:02.127225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.126798    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:02.130033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.130012    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.178352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.178317    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1685e-openshift-default-dockercfg-2cr54\""
Oct 31 16:16:02.181824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.181767    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.181978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.181872    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182027    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182079    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182202    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8plpb\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182232    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182298    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182345    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.182522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.182506    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.227363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.227326    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:16:02.283506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283455    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283518    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283548    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283571    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283598    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283662    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8plpb\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283692    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.283752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283721    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.283755    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.284001    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.284052    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.284126    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.284267    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.284585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.284558    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.286036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.286015    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.286287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.286268    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.336648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.336591    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.336837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.336749    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8plpb\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.444534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.444431    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:02.865906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.865874    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:16:02.868298 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:16:02.868263    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad1a287_7d7b_4979_ab94_124d1739061a.slice/crio-dc46e5d061e5c05d208384b50e0fa4dda73cb4fa343736047ebba32bb261e268 WatchSource:0}: Error finding container dc46e5d061e5c05d208384b50e0fa4dda73cb4fa343736047ebba32bb261e268: Status 404 returned error can't find the container with id dc46e5d061e5c05d208384b50e0fa4dda73cb4fa343736047ebba32bb261e268
Oct 31 16:16:02.871057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.871012    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:16:02.871149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.871108    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:16:02.871191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:02.871157    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:16:02.949664 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:16:02.949584    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:16:03.031186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.031132    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" event={"ID":"4ad1a287-7d7b-4979-ab94-124d1739061a","Type":"ContainerStarted","Data":"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"}
Oct 31 16:16:03.031388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.031200    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" event={"ID":"4ad1a287-7d7b-4979-ab94-124d1739061a","Type":"ContainerStarted","Data":"dc46e5d061e5c05d208384b50e0fa4dda73cb4fa343736047ebba32bb261e268"}
Oct 31 16:16:03.137888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.137767    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podStartSLOduration=1.1377334239999999 podStartE2EDuration="1.137733424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:03.137500401 +0000 UTC m=+808.199075589" watchObservedRunningTime="2026-10-31 16:16:03.137733424 +0000 UTC m=+808.199308611"
Oct 31 16:16:03.451700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.447694    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:03.451700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.448894    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:03.451700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:03.448959    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:04.447641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:04.447146    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:04.447641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:04.447223    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:05.446171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:05.446132    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:05.446501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:05.446453    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:06.454642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:06.450477    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:06.454642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:06.450566    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:07.450642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:07.448045    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:07.450642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:07.448116    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:08.460642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:08.460053    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 31 16:16:08.460642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:08.460121    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 31 16:16:09.604007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:09.603971    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:09.604660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:09.604575    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:09.605388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:09.605367    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:16:32.944884 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:16:32.944840    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:16:41.645547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.645513    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:16:41.646137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.646120    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e88015-631a-4f62-b14c-42e8fc2ab672" containerName="cleanup"
Oct 31 16:16:41.646182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.646141    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d5a945-17c2-4725-96dc-b1a049175adf" containerName="cleanup"
Oct 31 16:16:41.646182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.646148    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="36be6acb-a4b4-4f53-a32e-b02f44ec3d0a" containerName="cleanup"
Oct 31 16:16:41.649549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.649532    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.720460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.720424    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:16:41.743885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.743850    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"premium-w4-59357a-kserve-self-signed-certs\""
Oct 31 16:16:41.750530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750494    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t8b\" (UniqueName: \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.750696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750569    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.750696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750604    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.750820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750734    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.750820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750775    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.750922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.750852    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.851910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.851864    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t8b\" (UniqueName: \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.851925    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.851950    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.851993    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.852019    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.852085    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.852422    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.852453    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.852501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.852484    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.854407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.854377    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.854556 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.854539    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:41.904232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:41.904154    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t8b\" (UniqueName: \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\") pod \"premium-w4-59357a-kserve-5bbb8d4f6-mlsgs\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") " pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:42.015638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:42.013839    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:42.407444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:42.407403    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:16:42.409501 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:16:42.409471    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9fccb4f_41d9_4f92_b721_ecd3694120c6.slice/crio-5f8c3c17ba77d0664f74d00fbd984ee257f6906e8d836ae1a410fbb9ea4ae3dd WatchSource:0}: Error finding container 5f8c3c17ba77d0664f74d00fbd984ee257f6906e8d836ae1a410fbb9ea4ae3dd: Status 404 returned error can't find the container with id 5f8c3c17ba77d0664f74d00fbd984ee257f6906e8d836ae1a410fbb9ea4ae3dd
Oct 31 16:16:43.290346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:43.290309    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" event={"ID":"c9fccb4f-41d9-4f92-b721-ecd3694120c6","Type":"ContainerStarted","Data":"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"}
Oct 31 16:16:43.290346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:43.290359    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" event={"ID":"c9fccb4f-41d9-4f92-b721-ecd3694120c6","Type":"ContainerStarted","Data":"5f8c3c17ba77d0664f74d00fbd984ee257f6906e8d836ae1a410fbb9ea4ae3dd"}
Oct 31 16:16:43.397559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:43.397509    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podStartSLOduration=2.397494527 podStartE2EDuration="2.397494527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:43.395144377 +0000 UTC m=+848.456719563" watchObservedRunningTime="2026-10-31 16:16:43.397494527 +0000 UTC m=+848.459069712"
Oct 31 16:16:44.014564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:44.014527    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:52.014996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:52.014912    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:52.027636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:52.027590    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:16:52.366425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:16:52.366385    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:17:02.946085 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:17:02.946045    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:17:19.907708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:19.907666    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:17:19.908342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:19.907944    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy" containerID="cri-o://6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d" gracePeriod=30
Oct 31 16:17:25.180164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.180098    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:17:25.274274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274239    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.274470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274288    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.274470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274349    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.274470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274389    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.274470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274440    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274470    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274539    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274564    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274531    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274594    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\") pod \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\" (UID: \"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5\") "
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274664    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274890    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.275016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.274907    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275058    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275254    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275275    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275290    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275303    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.275446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.275317    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.276866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.276842    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:17:25.276866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.276862    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:25.277003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.276909    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:17:25.277160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.277144    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99" pod "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" (UID: "af3de8cc-4e7a-491d-99ba-b85e85b9e2b5"). InnerVolumeSpecName "kube-api-access-tlh99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:25.376876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.376833    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.376876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.376867    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlh99\" (UniqueName: \"kubernetes.io/projected/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-kube-api-access-tlh99\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.376876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.376882    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.377190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.376896    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:25.534244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.534148    4417 generic.go:356] "Generic (PLEG): container finished" podID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerID="6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d" exitCode=0
Oct 31 16:17:25.534244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.534202    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"
Oct 31 16:17:25.534531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.534236    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" event={"ID":"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5","Type":"ContainerDied","Data":"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"}
Oct 31 16:17:25.534531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.534292    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm" event={"ID":"af3de8cc-4e7a-491d-99ba-b85e85b9e2b5","Type":"ContainerDied","Data":"d44c2236d0a5a3768bd81ec798d2dfc28225317b511518565e1c6506a6ab202a"}
Oct 31 16:17:25.534531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.534316    4417 scope.go:122] "RemoveContainer" containerID="6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"
Oct 31 16:17:25.544078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.544061    4417 scope.go:122] "RemoveContainer" containerID="6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"
Oct 31 16:17:25.544326 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:17:25.544307    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d\": container with ID starting with 6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d not found: ID does not exist" containerID="6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"
Oct 31 16:17:25.544376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.544334    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d"} err="failed to get container status \"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d\": rpc error: code = NotFound desc = could not find container \"6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d\": container with ID starting with 6a57a17e640a369cb9bca16884d455123a272f2f0de4e038969bd3c01abb624d not found: ID does not exist"
Oct 31 16:17:25.685575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.685531    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:17:25.731326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:25.731278    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7c9qm"]
Oct 31 16:17:27.047957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:27.047921    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" path="/var/lib/kubelet/pods/af3de8cc-4e7a-491d-99ba-b85e85b9e2b5/volumes"
Oct 31 16:17:32.944656 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:17:32.944591    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:17:36.371047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.371005    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"]
Oct 31 16:17:36.371593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.371576    4417 cpu_manager.go:414] "containerMap: removing container" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy"
Oct 31 16:17:36.371667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.371594    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy"
Oct 31 16:17:36.371758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.371747    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3de8cc-4e7a-491d-99ba-b85e85b9e2b5" containerName="istio-proxy"
Oct 31 16:17:36.377121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.377100    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.429861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.429782    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"]
Oct 31 16:17:36.471592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.471561    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:36.471592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.471565    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:36.509079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.509049    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5fk\" (UniqueName: \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.509244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.509195    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.509298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.509259    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.610005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.609967    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5fk\" (UniqueName: \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.610188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.610061    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.610188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.610101    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.610710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.610689    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.612404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.612375    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.662272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.662206    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5fk\" (UniqueName: \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\") pod \"maas-api-revoke-sub-50d2c9e7-6mpxk\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:36.693073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:36.693048    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:37.059552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:37.059526    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"]
Oct 31 16:17:37.067204 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:17:37.067173    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a155e1_9701_42cf_8677_2f9df5818f8a.slice/crio-8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e WatchSource:0}: Error finding container 8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e: Status 404 returned error can't find the container with id 8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e
Oct 31 16:17:37.602324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:37.602238    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk" event={"ID":"95a155e1-9701-42cf-8677-2f9df5818f8a","Type":"ContainerStarted","Data":"76675e4958bd166ca2819221769825abc0b90b7f382aa6e9728f91d7b105a71d"}
Oct 31 16:17:37.602324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:37.602298    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk" event={"ID":"95a155e1-9701-42cf-8677-2f9df5818f8a","Type":"ContainerStarted","Data":"8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e"}
Oct 31 16:17:37.735160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:37.735106    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk" podStartSLOduration=1.735093596 podStartE2EDuration="1.735093596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:37.705832144 +0000 UTC m=+902.767407331" watchObservedRunningTime="2026-10-31 16:17:37.735093596 +0000 UTC m=+902.796668781"
Oct 31 16:17:38.609218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:38.609184    4417 generic.go:356] "Generic (PLEG): container finished" podID="95a155e1-9701-42cf-8677-2f9df5818f8a" containerID="76675e4958bd166ca2819221769825abc0b90b7f382aa6e9728f91d7b105a71d" exitCode=0
Oct 31 16:17:38.609663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:38.609273    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk" event={"ID":"95a155e1-9701-42cf-8677-2f9df5818f8a","Type":"ContainerDied","Data":"76675e4958bd166ca2819221769825abc0b90b7f382aa6e9728f91d7b105a71d"}
Oct 31 16:17:39.767415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.767382    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:39.949320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.949221    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\") pod \"95a155e1-9701-42cf-8677-2f9df5818f8a\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") "
Oct 31 16:17:39.949320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.949304    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\" (UniqueName: \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\") pod \"95a155e1-9701-42cf-8677-2f9df5818f8a\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") "
Oct 31 16:17:39.949692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.949516    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\") pod \"95a155e1-9701-42cf-8677-2f9df5818f8a\" (UID: \"95a155e1-9701-42cf-8677-2f9df5818f8a\") "
Oct 31 16:17:39.949971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.949937    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca" pod "95a155e1-9701-42cf-8677-2f9df5818f8a" (UID: "95a155e1-9701-42cf-8677-2f9df5818f8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:39.950101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.950079    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:39.951800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.951767    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk" pod "95a155e1-9701-42cf-8677-2f9df5818f8a" (UID: "95a155e1-9701-42cf-8677-2f9df5818f8a"). InnerVolumeSpecName "kube-api-access-xd5fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:39.952159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:39.952142    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca" pod "95a155e1-9701-42cf-8677-2f9df5818f8a" (UID: "95a155e1-9701-42cf-8677-2f9df5818f8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:40.051154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:40.051111    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5fk\" (UniqueName: \"kubernetes.io/projected/95a155e1-9701-42cf-8677-2f9df5818f8a-kube-api-access-xd5fk\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:40.051154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:40.051153    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a155e1-9701-42cf-8677-2f9df5818f8a-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:40.619956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:40.619919    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk" event={"ID":"95a155e1-9701-42cf-8677-2f9df5818f8a","Type":"ContainerDied","Data":"8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e"}
Oct 31 16:17:40.619956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:40.619933    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"
Oct 31 16:17:40.620220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:40.619969    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6adec857e081195051b51eb3c7747d21421f513316a2570585e1490ef8b01e"
Oct 31 16:17:54.288723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.288680    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"]
Oct 31 16:17:54.289366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.289344    4417 cpu_manager.go:414] "containerMap: removing container" podUID="95a155e1-9701-42cf-8677-2f9df5818f8a" containerName="revoke-keys"
Oct 31 16:17:54.289464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.289368    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a155e1-9701-42cf-8677-2f9df5818f8a" containerName="revoke-keys"
Oct 31 16:17:54.289533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.289518    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a155e1-9701-42cf-8677-2f9df5818f8a" containerName="revoke-keys"
Oct 31 16:17:54.292940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.292914    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.334845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.334813    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.334996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.334873    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxh5\" (UniqueName: \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.335067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.335045    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.387067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.387033    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:17:54.387243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.387163    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:17:54.389602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.389562    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"]
Oct 31 16:17:54.436603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.436571    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.436805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.436719    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.436805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.436767    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxh5\" (UniqueName: \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.437386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.437366    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.438987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.438965    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.489052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.489023    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxh5\" (UniqueName: \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\") pod \"maas-api-revoke-sub-505ac380-bqxhn\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.607061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.607025    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:54.978481 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:54.978452    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"]
Oct 31 16:17:54.981304 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:17:54.981265    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e01794a_9e0d_42d0_9462_4d30fdc4e4ba.slice/crio-fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919 WatchSource:0}: Error finding container fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919: Status 404 returned error can't find the container with id fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919
Oct 31 16:17:55.715784 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:55.715741    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn" event={"ID":"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba","Type":"ContainerStarted","Data":"452ce4750ad374a0661157a98209e67550d294b2cd3d3105ff716a84ba00c557"}
Oct 31 16:17:55.716199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:55.715806    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn" event={"ID":"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba","Type":"ContainerStarted","Data":"fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919"}
Oct 31 16:17:55.821677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:55.821594    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn" podStartSLOduration=1.821581482 podStartE2EDuration="1.821581482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:17:55.81982704 +0000 UTC m=+920.881402223" watchObservedRunningTime="2026-10-31 16:17:55.821581482 +0000 UTC m=+920.883156689"
Oct 31 16:17:56.723273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:56.723240    4417 generic.go:356] "Generic (PLEG): container finished" podID="8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" containerID="452ce4750ad374a0661157a98209e67550d294b2cd3d3105ff716a84ba00c557" exitCode=0
Oct 31 16:17:56.723788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:56.723332    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn" event={"ID":"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba","Type":"ContainerDied","Data":"452ce4750ad374a0661157a98209e67550d294b2cd3d3105ff716a84ba00c557"}
Oct 31 16:17:57.893324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.893295    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:57.992856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.992761    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\") pod \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") "
Oct 31 16:17:57.993024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.992905    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\") pod \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") "
Oct 31 16:17:57.993024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.992943    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\" (UniqueName: \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\") pod \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\" (UID: \"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba\") "
Oct 31 16:17:57.993127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.993082    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca" pod "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" (UID: "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:57.993319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.993297    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:57.994953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.994928    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca" pod "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" (UID: "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:57.995031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:57.994988    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5" pod "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" (UID: "8e01794a-9e0d-42d0-9462-4d30fdc4e4ba"). InnerVolumeSpecName "kube-api-access-6nxh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:58.094584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:58.094549    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:58.094584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:58.094581    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nxh5\" (UniqueName: \"kubernetes.io/projected/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba-kube-api-access-6nxh5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:17:58.734835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:58.734804    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"
Oct 31 16:17:58.735027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:58.734841    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn" event={"ID":"8e01794a-9e0d-42d0-9462-4d30fdc4e4ba","Type":"ContainerDied","Data":"fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919"}
Oct 31 16:17:58.735027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:17:58.734908    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc94c81b356d453c3cee3261479e4c2ba966ea33dd51a212b60f0a8a51328919"
Oct 31 16:18:02.944759 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:18:02.944720    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:18:04.931408 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.931372    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:18:04.932128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.932102    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" containerName="revoke-keys"
Oct 31 16:18:04.932128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.932127    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" containerName="revoke-keys"
Oct 31 16:18:04.932273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.932260    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" containerName="revoke-keys"
Oct 31 16:18:04.936138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.936111    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:04.989282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:04.989247    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:18:05.027379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.027329    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.027678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.027653    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.027826    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.027975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.027960    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.028125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.028081    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.028211 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.028139    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fxv\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.028273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.028241    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.028328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.028272    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.028385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.028337    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129265    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129315    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129372    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129407    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129441    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129487    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129523    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129568    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129656    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxv\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129866    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.129972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.129891    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.130102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.130026    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.130102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.130084    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.130351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.130326    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.131912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.131886    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.132076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.132059    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.182803 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.182727    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.182931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.182875    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fxv\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jbh6n\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.251038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.251003    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:05.669775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.669745    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:18:05.672154 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:18:05.672123    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9d2fe5_7a8c_4c4d_95ee_ec10bc56b111.slice/crio-808552477a7e0b68bb8e6aad859b0ca780288f74f24f18d745bbb1f71e5687c1 WatchSource:0}: Error finding container 808552477a7e0b68bb8e6aad859b0ca780288f74f24f18d745bbb1f71e5687c1: Status 404 returned error can't find the container with id 808552477a7e0b68bb8e6aad859b0ca780288f74f24f18d745bbb1f71e5687c1
Oct 31 16:18:05.674793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.674748    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.674884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.674843    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.674924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.674890    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:05.793429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.793387    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" event={"ID":"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111","Type":"ContainerStarted","Data":"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"}
Oct 31 16:18:05.793597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.793466    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" event={"ID":"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111","Type":"ContainerStarted","Data":"808552477a7e0b68bb8e6aad859b0ca780288f74f24f18d745bbb1f71e5687c1"}
Oct 31 16:18:05.907514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:05.907452    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podStartSLOduration=1.9074350629999999 podStartE2EDuration="1.907435063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:05.904511625 +0000 UTC m=+930.966086811" watchObservedRunningTime="2026-10-31 16:18:05.907435063 +0000 UTC m=+930.969010248"
Oct 31 16:18:06.256640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:06.253066    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:06.260641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:06.258770    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:06.260641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:06.258856    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:07.255492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.255024    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:07.255492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.255090    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:07.484260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.484224    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"]
Oct 31 16:18:07.488453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.488428    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.574217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.573562    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:18:07.574217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.573748    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:18:07.576009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.575966    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.576152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.576132    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.576325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.576302    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqtsg\" (UniqueName: \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.577692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.577669    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"]
Oct 31 16:18:07.677092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.677057    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqtsg\" (UniqueName: \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.677303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.677100    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.677303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.677173    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.677983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.677961    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.679666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.679605    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.730670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.730634    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqtsg\" (UniqueName: \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\") pod \"maas-api-revoke-sub-8b2bb2b5-7xbv2\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:07.803004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:07.802961    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:08.199752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.199726    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"]
Oct 31 16:18:08.201906 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:18:08.201870    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9e4d2a_5e94_46be_9c92_3d982be6bc2a.slice/crio-c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60 WatchSource:0}: Error finding container c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60: Status 404 returned error can't find the container with id c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60
Oct 31 16:18:08.259644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.258052    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:08.259644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.258152    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:08.814439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.814399    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2" event={"ID":"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a","Type":"ContainerStarted","Data":"ea9d6109cadc1cdd5c414b81837771cd4de9fcd18c705bbeff4f5ebe903b07a0"}
Oct 31 16:18:08.814875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.814464    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2" event={"ID":"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a","Type":"ContainerStarted","Data":"c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60"}
Oct 31 16:18:08.921300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:08.921247    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2" podStartSLOduration=1.921230991 podStartE2EDuration="1.921230991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:08.918418984 +0000 UTC m=+933.979994169" watchObservedRunningTime="2026-10-31 16:18:08.921230991 +0000 UTC m=+933.982806176"
Oct 31 16:18:09.256213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:09.256112    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:09.256478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:09.256449    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:09.821600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:09.821565    4417 generic.go:356] "Generic (PLEG): container finished" podID="3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" containerID="ea9d6109cadc1cdd5c414b81837771cd4de9fcd18c705bbeff4f5ebe903b07a0" exitCode=0
Oct 31 16:18:09.822029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:09.821604    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2" event={"ID":"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a","Type":"ContainerDied","Data":"ea9d6109cadc1cdd5c414b81837771cd4de9fcd18c705bbeff4f5ebe903b07a0"}
Oct 31 16:18:10.260703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:10.260135    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:10.260703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:10.260216    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:11.009863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.009832    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:11.114956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.114866    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\") pod \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") "
Oct 31 16:18:11.115114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.115094    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\" (UniqueName: \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\") pod \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") "
Oct 31 16:18:11.115153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.115130    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\") pod \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\" (UID: \"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a\") "
Oct 31 16:18:11.115527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.115497    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca" pod "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" (UID: "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:11.117551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.117526    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca" pod "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" (UID: "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:18:11.117675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.117570    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg" pod "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" (UID: "3c9e4d2a-5e94-46be-9c92-3d982be6bc2a"). InnerVolumeSpecName "kube-api-access-lqtsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:11.216359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.216315    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:11.216359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.216345    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqtsg\" (UniqueName: \"kubernetes.io/projected/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-kube-api-access-lqtsg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:11.216359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.216355    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:18:11.257594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.257097    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:11.257594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.257167    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:11.844165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.844119    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2" event={"ID":"3c9e4d2a-5e94-46be-9c92-3d982be6bc2a","Type":"ContainerDied","Data":"c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60"}
Oct 31 16:18:11.844165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.844164    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"
Oct 31 16:18:11.844433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:11.844216    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80b0ee3db1a891971a113382ca57a7c534024501b861b91581acc2d85334b60"
Oct 31 16:18:12.252246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:12.252155    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:12.252644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:12.252240    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:13.251708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:13.251668    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jbh6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 16:18:13.251927 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:13.251769    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 16:18:14.255903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:14.255872    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:14.256403 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:14.256292    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:14.257182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:14.257153    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:18:18.155466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.155371    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:18:18.155968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.155891    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" containerName="revoke-keys"
Oct 31 16:18:18.155968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.155903    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" containerName="revoke-keys"
Oct 31 16:18:18.156087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.156004    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" containerName="revoke-keys"
Oct 31 16:18:18.159992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.159968    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226083    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226146    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226252    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226290    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226314    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226417    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226469    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226506    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.226654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.226602    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjhf\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.248522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.248480    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:18:18.328031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.327994    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328055    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjhf\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328097    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328142    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328165    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328185    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328211    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328245    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328354    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328497    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328696    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328861    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.328980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.328869    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.329099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.329078    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.330689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.330667    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.331017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.330999    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.384069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.384039    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.384555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.384528    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjhf\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.473582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.473494    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:18.775453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.775342    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:18:18.781032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.781003    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.833116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.833083    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:18:18.834465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834438    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834526    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834584    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834651    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834694    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndbj\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834736    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834761    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834800    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.834965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.834843    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.928182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.928155    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:18:18.930196 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:18:18.930168    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01dac0c_4523_4f5a_b18a_653d540abae0.slice/crio-3e8b4984da62ae9bc0f8314269598b0364fc7ef317c98a34700bf6ed0b0ecf27 WatchSource:0}: Error finding container 3e8b4984da62ae9bc0f8314269598b0364fc7ef317c98a34700bf6ed0b0ecf27: Status 404 returned error can't find the container with id 3e8b4984da62ae9bc0f8314269598b0364fc7ef317c98a34700bf6ed0b0ecf27
Oct 31 16:18:18.935543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935517    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.935670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935571    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.935670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935604    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.935670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935606    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:18.935842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935696    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:18.935842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935744    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:18.935996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935970    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.935996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.935986    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936026    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936064    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndbj\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936100    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936126    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936162    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936201    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936262    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936535    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.936929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.936908    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.938118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.938074    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.938597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.938580    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.989154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.989125    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:18.989447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:18.989419    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndbj\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:19.071078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.071036    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:18:19.075515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.075487    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.095959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.095932    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:19.126283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.124526    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-59357a-openshift-default-dockercfg-z7nvh\""
Oct 31 16:18:19.132123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.130154    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:18:19.145678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145638    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.145847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145687    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.145847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145738    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.145847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145798    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.146006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145844    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.146006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145885    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrc7\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.146006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.145947    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.146158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.146004    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.146158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.146034    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247042    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247185    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247214    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247261    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247312    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247359    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247408    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrc7\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247473    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247514    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.247987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.247963    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.249082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.248768    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.249082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.248817    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.249469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.249444    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.249730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.249691    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.251049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.251023    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.251682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.251664    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.300704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.300671    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.300856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.300734    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrc7\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.393267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.393179    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:19.486643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.486587    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:18:19.489414 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:18:19.489370    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408aa05b_4a02_4177_ad74_ffa930bd8dc2.slice/crio-cb4f8657660831c308874a1a22e31ae8833a665ae6a61810bb1690ce8a36b7a3 WatchSource:0}: Error finding container cb4f8657660831c308874a1a22e31ae8833a665ae6a61810bb1690ce8a36b7a3: Status 404 returned error can't find the container with id cb4f8657660831c308874a1a22e31ae8833a665ae6a61810bb1690ce8a36b7a3
Oct 31 16:18:19.492468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.492424    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.492588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.492530    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.492588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.492577    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.786139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.786096    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.786276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.786182    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.786276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.786229    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:19.786915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.786867    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:18:19.927214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.927176    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" event={"ID":"1804c48b-5202-48a6-be6e-06b0307f2ba8","Type":"ContainerStarted","Data":"7bb603ab0297ac53296774170888e5c6ac19df88f74ee2043e11dfc1aea275c5"}
Oct 31 16:18:19.929834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.929212    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" event={"ID":"408aa05b-4a02-4177-ad74-ffa930bd8dc2","Type":"ContainerStarted","Data":"a762102448a8a0e16610cad924b0e6202a3b8acef27c234df841e760cff3db0a"}
Oct 31 16:18:19.929834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.929269    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" event={"ID":"408aa05b-4a02-4177-ad74-ffa930bd8dc2","Type":"ContainerStarted","Data":"cb4f8657660831c308874a1a22e31ae8833a665ae6a61810bb1690ce8a36b7a3"}
Oct 31 16:18:19.931923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.931898    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" event={"ID":"f01dac0c-4523-4f5a-b18a-653d540abae0","Type":"ContainerStarted","Data":"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"}
Oct 31 16:18:19.932060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.931942    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" event={"ID":"f01dac0c-4523-4f5a-b18a-653d540abae0","Type":"ContainerStarted","Data":"3e8b4984da62ae9bc0f8314269598b0364fc7ef317c98a34700bf6ed0b0ecf27"}
Oct 31 16:18:20.017750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:19.995692    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:18:20.105779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.105742    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:20.105952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.105794    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:18:20.112527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.112500    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:20.112770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.112744    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:20.112884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.112755    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.113684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.113601    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podStartSLOduration=2.113584306 podStartE2EDuration="2.113584306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:20.056066553 +0000 UTC m=+945.117641741" watchObservedRunningTime="2026-10-31 16:18:20.113584306 +0000 UTC m=+945.175159493"
Oct 31 16:18:20.117579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.117546    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:18:20.117720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.117604    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:18:20.117781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.117725    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.178173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.178104    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podStartSLOduration=2.178082975 podStartE2EDuration="2.178082975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:20.175172896 +0000 UTC m=+945.236748082" watchObservedRunningTime="2026-10-31 16:18:20.178082975 +0000 UTC m=+945.239658163"
Oct 31 16:18:20.207545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207508    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.207545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207549    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.207812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207565    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.207812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207709    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.207812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207760    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.207812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207827    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.207922    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208004    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208038    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208124    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208161    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208194    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6g6\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208287    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208313    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208334    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.208421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208371    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ffg\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.208631 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.208447    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310368 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310319    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310415    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310454    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310480    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6g6\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310519    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310548    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310580    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310665    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ffg\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310699    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310790    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310790    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310817    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310842    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.310876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310886    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310902    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310947    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310956    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.310984    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311012    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311068    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311086    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311374    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311472    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311516    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311521    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.311557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.311519    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.312193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.312148    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.312249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.312229    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.313554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.313531    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.313678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.313584    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.313723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.313683    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.314154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.314137    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.389856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.381513    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.389856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.383201    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.389856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.383921    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ffg\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-h8jrx\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.392675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.390771    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6g6\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s2qtj\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.430508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.430274    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:20.477722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.475580    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:20.477722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.475769    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:20.477722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.475816    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:20.477722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.476355    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:20.994086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:20.970699    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:18:21.084105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.080653    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.084105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.080752    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.084105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.080797    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.114964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.103556    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:21.114964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.103658    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:21.419654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.413724    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:18:21.419654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.414711    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.419654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.414806    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.419654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.414856    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:21.456686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.437159    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" event={"ID":"1804c48b-5202-48a6-be6e-06b0307f2ba8","Type":"ContainerStarted","Data":"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"}
Oct 31 16:18:21.456686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.454755    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:21.486650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.474984    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:21.486650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.475047    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:21.505653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.500777    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:21.505653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:21.500843    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:22.098013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.097973    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.098270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.098240    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:22.148649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.142284    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" event={"ID":"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b","Type":"ContainerStarted","Data":"31db2d59016bfc18e5e15e960d78afac6f21b2a1033d07f76d88064c21ff2aaf"}
Oct 31 16:18:22.148950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.148918    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" event={"ID":"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b","Type":"ContainerStarted","Data":"1866d466ea1700779049c151ef2b3df835582dc49c5f0ed5525e5183bbc8568b"}
Oct 31 16:18:22.178639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.161921    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" event={"ID":"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca","Type":"ContainerStarted","Data":"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"}
Oct 31 16:18:22.181086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.181053    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" event={"ID":"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca","Type":"ContainerStarted","Data":"ae2935eb68da99a06333cb781c926fec9d6fcce9a8843e78babfcd18aacd1f72"}
Oct 31 16:18:22.269834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.269755    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podStartSLOduration=3.269732576 podStartE2EDuration="3.269732576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:21.491775309 +0000 UTC m=+946.553350478" watchObservedRunningTime="2026-10-31 16:18:22.269732576 +0000 UTC m=+947.331307764"
Oct 31 16:18:22.295176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.295105    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podStartSLOduration=3.295083028 podStartE2EDuration="3.295083028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:22.280042955 +0000 UTC m=+947.341618142" watchObservedRunningTime="2026-10-31 16:18:22.295083028 +0000 UTC m=+947.356658215"
Oct 31 16:18:22.395987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.395883    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.396276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.396229    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:22.417571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.417503    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podStartSLOduration=3.417480821 podStartE2EDuration="3.417480821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:22.413809104 +0000 UTC m=+947.475384340" watchObservedRunningTime="2026-10-31 16:18:22.417480821 +0000 UTC m=+947.479056009"
Oct 31 16:18:22.456828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.456793    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:22.460799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.460757    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.460965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.460834    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:22.482491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.482458    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:22.489274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.489231    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.489513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.489485    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:22.489757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.489740    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:22.489869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:22.489850    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:23.109099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.109056    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.109377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.109337    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:23.394177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.394075    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.394453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.394420    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:23.436083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.436042    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.436334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.436303    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:23.477113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.477071    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.478057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.478024    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:23.478204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.477963    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:23.478319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:23.478302    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:24.111195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.111157    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.111548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.111488    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:24.397193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.397095    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.397507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.397477    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:24.433049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.433010    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.433344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.433313    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:24.482487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.481336    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.482487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.481397    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:24.482487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.482377    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:24.482487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:24.482419    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:25.166639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.163199    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.166639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.163263    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:25.447038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.446941    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.447600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.447569    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:25.468189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.447496    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.468375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.468343    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:25.476120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.476076    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.476354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.476328    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:25.477100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.477078    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:25.477264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:25.477243    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:26.166643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.148365    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.257809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.238048    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:26.406166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.406123    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.406462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.406427    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:26.455703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.446239    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.455703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.446317    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:26.476101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.476058    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.476417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.476385    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:26.481023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.480989    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:26.481217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:26.481190    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:27.099132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.099092    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.099413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.099372    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.500806    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.500876    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.503845    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.503893    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.504122    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.504153    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.504372    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.504403    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 16:18:27.507702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:27.486723    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:18:28.098093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.098049    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 16:18:28.098525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.098492    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 16:18:28.405709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.404923    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 16:18:28.405709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.404988    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 16:18:28.445977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.445935    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:28.447097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.447060    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:28.478985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.478947    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:28.479264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.479234    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:28.599873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.599835    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:28.610136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.601357    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:28.610136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:28.609988    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:18:29.304324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.304287    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:29.306253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.306117    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:29.307539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.307522    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:18:29.430744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.430703    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:29.430921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.430780    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:29.494107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.494065    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:29.494406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.494373    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:29.524239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.522830    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:29.524239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.523685    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:29.527225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:29.527202    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:18:30.430818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:30.430775    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:30.431346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:30.430825    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:30.477302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:30.477256    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:30.477503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:30.477325    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:31.431364 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:31.431317    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s2qtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 31 16:18:31.431909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:31.431393    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 31 16:18:31.478017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:31.477972    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-h8jrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 16:18:31.478183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:31.478041    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 16:18:32.493833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.493800    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:32.495559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.495529    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:32.496889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.496866    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:18:32.805560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.804992    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:32.807144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.806198    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:32.807566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:32.807534    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:18:32.945580 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:18:32.945543    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:18:49.659337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.659300    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:18:49.663690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.663667    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.711538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.711506    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7f13a-openshift-default-dockercfg-m7lj5\""
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782126    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782191    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782299    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782335    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782435    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782511    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4brx\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782545    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.788739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.782632    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.808053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.808015    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:18:49.883633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883593    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883663    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883719    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4brx\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883745    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883784    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883806    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.883840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883831    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883927    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.883966    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.884090    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.884395    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.884481    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.884505    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.884716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.884691    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.886247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.886222    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.886406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.886383    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.937943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.937875    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4brx\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.937943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.937877    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:49.978163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:49.978121    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:50.456876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.456832    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:18:50.459515 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:18:50.459468    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72307cd0_e3bf_48df_b4eb_890b9d3a4ad4.slice/crio-518327e1c4ddfd84bef96c24caaee2c9723c3ca76d2e3b1ef74ca681ab17b9de WatchSource:0}: Error finding container 518327e1c4ddfd84bef96c24caaee2c9723c3ca76d2e3b1ef74ca681ab17b9de: Status 404 returned error can't find the container with id 518327e1c4ddfd84bef96c24caaee2c9723c3ca76d2e3b1ef74ca681ab17b9de
Oct 31 16:18:50.464647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.462660    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:50.464647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.462836    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:50.464647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.462957    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:18:50.753948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.753858    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" event={"ID":"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4","Type":"ContainerStarted","Data":"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"}
Oct 31 16:18:50.754339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.753974    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" event={"ID":"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4","Type":"ContainerStarted","Data":"518327e1c4ddfd84bef96c24caaee2c9723c3ca76d2e3b1ef74ca681ab17b9de"}
Oct 31 16:18:50.861953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.861891    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podStartSLOduration=1.861878927 podStartE2EDuration="1.861878927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:50.861113405 +0000 UTC m=+975.922688588" watchObservedRunningTime="2026-10-31 16:18:50.861878927 +0000 UTC m=+975.923454112"
Oct 31 16:18:50.978919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:50.978878    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:51.980511 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:51.980473    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:51.980997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:51.980553    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:52.978918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:52.978882    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:52.979093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:52.978954    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:53.979031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:53.978992    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:53.979455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:53.979053    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:54.979062 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:54.979024    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:54.979499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:54.979082    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:55.979393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:55.979354    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:55.979831 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:55.979423    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:56.979379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:56.979326    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 16:18:56.979591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:56.979386    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 16:18:57.983540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:57.983505    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:57.984094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:57.984070    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:18:57.984747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:18:57.984722    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:19:02.945512 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:19:02.945471    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:19:13.241055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.241018    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:19:13.245157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.245131    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.299063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.298988    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:19:13.316987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.316951    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.317176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.317070    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.317176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.317143    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.317176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.317164    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.317391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.317366    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.317492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.317472    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfhj8\" (UniqueName: \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.340218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.340180    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"openshift-service-ca.crt\""
Oct 31 16:19:13.340397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.340352    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"kube-root-ca.crt\""
Oct 31 16:19:13.340397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.340368    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"default-dockercfg-8rxfz\""
Oct 31 16:19:13.340891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.340871    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"distinct-w3-f7821a-kserve-self-signed-certs\""
Oct 31 16:19:13.418512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.418474    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.428267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.421945    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.428688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.428658    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.428891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.428873    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.429195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.429167    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.434786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.434755    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfhj8\" (UniqueName: \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.435260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.435240    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.440646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.439043    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.440646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.440104    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.440646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.440352    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.440883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.440696    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.487349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.487319    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfhj8\" (UniqueName: \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\") pod \"distinct-w3-f7821a-kserve-57b94d7fc7-swc9w\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") " pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:13.559521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:13.559483    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:14.007937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:14.007908    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:19:14.009323 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:19:14.009297    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aac4bdf_d175_4447_ae7c_3b991ac5703e.slice/crio-4e25383e677215ab6c7cbf795f0ba6cd5d60410a3fc628eb2b697aa2197f2228 WatchSource:0}: Error finding container 4e25383e677215ab6c7cbf795f0ba6cd5d60410a3fc628eb2b697aa2197f2228: Status 404 returned error can't find the container with id 4e25383e677215ab6c7cbf795f0ba6cd5d60410a3fc628eb2b697aa2197f2228
Oct 31 16:19:14.972459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:14.972415    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" event={"ID":"5aac4bdf-d175-4447-ae7c-3b991ac5703e","Type":"ContainerStarted","Data":"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"}
Oct 31 16:19:14.972895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:14.972475    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" event={"ID":"5aac4bdf-d175-4447-ae7c-3b991ac5703e","Type":"ContainerStarted","Data":"4e25383e677215ab6c7cbf795f0ba6cd5d60410a3fc628eb2b697aa2197f2228"}
Oct 31 16:19:15.078473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:15.078425    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podStartSLOduration=2.078411277 podStartE2EDuration="2.078411277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:15.077013996 +0000 UTC m=+1000.138589179" watchObservedRunningTime="2026-10-31 16:19:15.078411277 +0000 UTC m=+1000.139986499"
Oct 31 16:19:15.559964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:15.559922    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:23.560673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:23.560639    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:23.573790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:23.573759    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:24.036493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:24.036400    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:19:32.946290 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:19:32.946251    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:19:34.374576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.374536    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:19:34.378689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.378665    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.474536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.474496    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:19:34.547947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.547917    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.547976    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548008    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548098    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548162    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548222    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548319    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548408 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548395    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlph\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.548518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.548502    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649204    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649248    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649277    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649359    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649411    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649449    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649506    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlph\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649646    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649658    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.649839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.649715    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.650329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.650301    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.650463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.650417    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.650463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.650448    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.650802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.650781    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.652246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.652226    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.652745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.652721    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.704161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.703712    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.705737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.705702    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlph\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:34.994320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:34.994237    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:35.418957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:35.418930    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:19:35.421016 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:19:35.420982    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce2dce3_4cfd_476b_b812_1c0926b8b88a.slice/crio-1dc806f339da90f75d8601fb32c91e106f826c1905b628caca23b22f6a4990a1 WatchSource:0}: Error finding container 1dc806f339da90f75d8601fb32c91e106f826c1905b628caca23b22f6a4990a1: Status 404 returned error can't find the container with id 1dc806f339da90f75d8601fb32c91e106f826c1905b628caca23b22f6a4990a1
Oct 31 16:19:35.423398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:35.423369    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:35.423475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:35.423429    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:35.423475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:35.423458    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:36.160256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:36.160222    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" event={"ID":"8ce2dce3-4cfd-476b-b812-1c0926b8b88a","Type":"ContainerStarted","Data":"1ce30bdf30b54ff325acdc7c06ff214378f7802f10db68cc4b890598e324757f"}
Oct 31 16:19:36.160434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:36.160288    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" event={"ID":"8ce2dce3-4cfd-476b-b812-1c0926b8b88a","Type":"ContainerStarted","Data":"1dc806f339da90f75d8601fb32c91e106f826c1905b628caca23b22f6a4990a1"}
Oct 31 16:19:36.994552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:36.994513    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:36.996200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:36.996174    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:36.996330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:36.996224    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:37.994997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:37.994960    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:37.995385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:37.995012    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:38.994719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:38.994688    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:38.994890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:38.994738    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:39.995462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:39.995425    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:39.995869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:39.995476    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:40.995241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:40.995203    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:40.995417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:40.995254    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:41.994812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:41.994774    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 16:19:41.995275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:41.994840    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 16:19:42.999997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:42.999968    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:43.000440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:43.000237    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:43.001035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:43.001017    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:19:43.206713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:43.206663    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podStartSLOduration=9.206648934 podStartE2EDuration="9.206648934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:36.360912492 +0000 UTC m=+1021.422487679" watchObservedRunningTime="2026-10-31 16:19:43.206648934 +0000 UTC m=+1028.268224120"
Oct 31 16:19:48.834488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.834408    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:19:48.839017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.838994    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901158    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901213    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901365    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901408    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901513    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901566    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qjf\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901705    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901803    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.901923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.901850    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:48.937416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:48.937385    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:19:49.002861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.002827    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.002861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.002864    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.002913    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.002931    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.002954    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003103    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qjf\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003189    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003224    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003271    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003284    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003350    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003541    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003724    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.003891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.003805    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.005322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.005299    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.005423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.005407    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.060252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.060205    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qjf\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.060425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.060270    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.092142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.092059    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:19:49.097401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.097380    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.151808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.151780    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:19:49.153390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.153360    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206050    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206093    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd96s\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206148    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206177    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206209    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206241    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206308    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.207316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.206455    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307379    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307427    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307453    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307473    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307517    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307575    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307608    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307790    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.307878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.307814    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd96s\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.308035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.308011    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.308141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.308109    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.308235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.308210    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.308277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.308260    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.308374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.308358    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.309875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.309857    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.310080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.310064    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.360499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.360424    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd96s\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.360941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.360918    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.411399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.411365    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:49.575200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.575172    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:19:49.576746 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:19:49.576719    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77980ac3_7ec2_475b_8191_a6988512ae66.slice/crio-9d8535d8b70e7974de6e6ffbffc4bb314386afcdc46145ceb7b4d3fec5c77319 WatchSource:0}: Error finding container 9d8535d8b70e7974de6e6ffbffc4bb314386afcdc46145ceb7b4d3fec5c77319: Status 404 returned error can't find the container with id 9d8535d8b70e7974de6e6ffbffc4bb314386afcdc46145ceb7b4d3fec5c77319
Oct 31 16:19:49.579255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.579224    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.579327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.579286    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.579327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.579315    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.783537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.783483    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:19:49.784901 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:19:49.784869    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58203f05_1714_450a_a50c_42592d06eb5f.slice/crio-22259e96f40b8ff94b1b50c0905580619ee52f07dbe6387fea3df3a2034ebe0a WatchSource:0}: Error finding container 22259e96f40b8ff94b1b50c0905580619ee52f07dbe6387fea3df3a2034ebe0a: Status 404 returned error can't find the container with id 22259e96f40b8ff94b1b50c0905580619ee52f07dbe6387fea3df3a2034ebe0a
Oct 31 16:19:49.788074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.788042    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.788162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.788114    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:49.788162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:49.788145    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:19:50.262370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.262330    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" event={"ID":"77980ac3-7ec2-475b-8191-a6988512ae66","Type":"ContainerStarted","Data":"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"}
Oct 31 16:19:50.262893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.262411    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" event={"ID":"77980ac3-7ec2-475b-8191-a6988512ae66","Type":"ContainerStarted","Data":"9d8535d8b70e7974de6e6ffbffc4bb314386afcdc46145ceb7b4d3fec5c77319"}
Oct 31 16:19:50.265146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.265118    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" event={"ID":"58203f05-1714-450a-a50c-42592d06eb5f","Type":"ContainerStarted","Data":"50d76eafa7a1a762b80e10038acd1cf10ff44c51b5d7d3d024d5f0ed1747c710"}
Oct 31 16:19:50.265298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.265160    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" event={"ID":"58203f05-1714-450a-a50c-42592d06eb5f","Type":"ContainerStarted","Data":"22259e96f40b8ff94b1b50c0905580619ee52f07dbe6387fea3df3a2034ebe0a"}
Oct 31 16:19:50.371042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.370968    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podStartSLOduration=2.370950514 podStartE2EDuration="2.370950514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.368247777 +0000 UTC m=+1035.429822963" watchObservedRunningTime="2026-10-31 16:19:50.370950514 +0000 UTC m=+1035.432525706"
Oct 31 16:19:50.412288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.412259    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:50.413750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.413727    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:50.413807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.413777    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:50.477648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:50.477545    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podStartSLOduration=1.477526502 podStartE2EDuration="1.477526502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.475464506 +0000 UTC m=+1035.537039698" watchObservedRunningTime="2026-10-31 16:19:50.477526502 +0000 UTC m=+1035.539101691"
Oct 31 16:19:51.157654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:51.155683    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:51.185646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:51.184852    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:51.185646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:51.184938    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:51.412707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:51.412596    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:51.413176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:51.412948    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:52.155057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:52.155016    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:52.155321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:52.155290    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:52.412096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:52.412008    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:52.412096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:52.412057    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:53.163641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:53.163046    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:53.163641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:53.163110    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:53.411881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:53.411848    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:53.412050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:53.411899    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:54.160642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:54.159739    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:54.160642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:54.159797    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:54.419714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:54.412694    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:54.419714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:54.412759    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:55.195642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:55.165999    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:55.195642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:55.166065    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:55.411809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:55.411775    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:55.412004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:55.411836    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:56.164744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:56.164704    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 16:19:56.165336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:56.165297    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 16:19:56.412045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:56.412006    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:19:56.412224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:56.412083    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:19:57.445640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.444750    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:57.446446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.446406    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:57.453502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.453480    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:19:57.789076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.788997    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:57.790413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.790375    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:19:57.793537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:19:57.793516    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:20:02.944702 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:20:02.944663    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:20:19.945729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:19.945691    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:20:19.950742 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:19.950720    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.051202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.051156    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:20:20.099909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.099873    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.099925    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100038    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gk7b\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100096    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100169    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100263    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100306    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100343    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.100446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.100410    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201402    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201471    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201497    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201516    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201551    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.201777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201738    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201779    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201825    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gk7b\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201901    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201956    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.201978    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.202229    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.202446    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.202595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.202573    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.203978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.203954    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.204217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.204201    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.254464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.254435    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.254702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.254679    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gk7b\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.265436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.265411    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:20.688579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.688534    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:20:20.690221 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:20:20.690187    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cec1a0_366f_4a49_bea0_d84997b68680.slice/crio-fc8398a550dc7da3435850ef160cee78bdac9d91f799f86c8083a5061818e9e0 WatchSource:0}: Error finding container fc8398a550dc7da3435850ef160cee78bdac9d91f799f86c8083a5061818e9e0: Status 404 returned error can't find the container with id fc8398a550dc7da3435850ef160cee78bdac9d91f799f86c8083a5061818e9e0
Oct 31 16:20:20.692724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.692687    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:20.692829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.692767    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:20.692829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:20.692816    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:21.585704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:21.585658    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" event={"ID":"31cec1a0-366f-4a49-bea0-d84997b68680","Type":"ContainerStarted","Data":"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"}
Oct 31 16:20:21.586131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:21.585729    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" event={"ID":"31cec1a0-366f-4a49-bea0-d84997b68680","Type":"ContainerStarted","Data":"fc8398a550dc7da3435850ef160cee78bdac9d91f799f86c8083a5061818e9e0"}
Oct 31 16:20:21.694236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:21.694184    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podStartSLOduration=2.6941689650000002 podStartE2EDuration="2.694168965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:21.69287908 +0000 UTC m=+1066.754454260" watchObservedRunningTime="2026-10-31 16:20:21.694168965 +0000 UTC m=+1066.755744150"
Oct 31 16:20:22.266066 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:22.265994    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:22.268216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:22.268181    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:22.268371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:22.268245    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:23.266493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:23.266458    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:23.266910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:23.266516    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:24.266729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:24.266685    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:24.267165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:24.266756    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:25.266171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:25.266129    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:25.266388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:25.266189    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:26.266183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:26.266149    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:26.266611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:26.266208    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:27.266299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:27.266259    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 16:20:27.266715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:27.266325    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 16:20:28.270553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:28.270518    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:28.270954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:28.270922    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:28.271726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:28.271703    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:20:32.945153 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:20:32.945116    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:20:35.053876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.053842    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:20:35.058738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.058714    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.138278 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.137961    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.138469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.138282    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.138556 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.138449    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.138829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.138799    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb57r\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.139057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.139041    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.139184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.139169    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.139316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.139302    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.139420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.139406    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.139540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.139527    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.158132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.158095    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:20:35.241279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241246    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241284    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb57r\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241328    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241353    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241387    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241417    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241465    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241563    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241589    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.241966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241902    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.242025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241964    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.242025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.241999    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.242406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.242333    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.242529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.242438    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.244643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.244587    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.244758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.244741    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.295006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.294965    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.295161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.295094    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb57r\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.374952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.374844    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:35.795393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.795363    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:20:35.797167 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:20:35.797138    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1d880b_ff80_4de9_b6ec_eef487720152.slice/crio-01a380fe124ce5a9fb96c18cb06b6afe4c48251e777112110d49d8438e7fb8ba WatchSource:0}: Error finding container 01a380fe124ce5a9fb96c18cb06b6afe4c48251e777112110d49d8438e7fb8ba: Status 404 returned error can't find the container with id 01a380fe124ce5a9fb96c18cb06b6afe4c48251e777112110d49d8438e7fb8ba
Oct 31 16:20:35.799725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.799692    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:35.799799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.799760    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:35.799840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:35.799810    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:20:36.681375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:36.681335    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" event={"ID":"1d1d880b-ff80-4de9-b6ec-eef487720152","Type":"ContainerStarted","Data":"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"}
Oct 31 16:20:36.681813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:36.681438    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" event={"ID":"1d1d880b-ff80-4de9-b6ec-eef487720152","Type":"ContainerStarted","Data":"01a380fe124ce5a9fb96c18cb06b6afe4c48251e777112110d49d8438e7fb8ba"}
Oct 31 16:20:37.378060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:37.378025    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:37.391217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:37.391168    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:37.391595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:37.391558    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:38.376100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:38.376060    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:38.376695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:38.376661    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:39.375570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:39.375531    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:39.375769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:39.375601    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:40.375800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:40.375766    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:40.376190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:40.375825    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:41.376171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:41.376131    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:41.376599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:41.376203    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:42.375670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:42.375600    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 16:20:42.375853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:42.375699    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 16:20:43.380255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:43.380223    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:43.380778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:43.380507    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:43.381204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:43.381186    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:20:43.534802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:20:43.534747    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podStartSLOduration=8.534734148 podStartE2EDuration="8.534734148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:36.840880917 +0000 UTC m=+1081.902456102" watchObservedRunningTime="2026-10-31 16:20:43.534734148 +0000 UTC m=+1088.596309334"
Oct 31 16:21:02.944604 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:21:02.944567    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:21:09.740906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.740865    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:21:09.745719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.745694    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.797461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.797430    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"openshift-service-ca.crt\""
Oct 31 16:21:09.818593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.818550    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22vh\" (UniqueName: \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.818827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.818742    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.818910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.818877    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.818995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.818969    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.819083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.819009    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.819083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.819045    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.843132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.843100    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"default-dockercfg-jbp9j\""
Oct 31 16:21:09.843431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.843415    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"distinct-2-w1-4fe3a4-kserve-self-signed-certs\""
Oct 31 16:21:09.843683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.843667    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"kube-root-ca.crt\""
Oct 31 16:21:09.848560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.848532    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930154    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930209    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930241    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930322    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22vh\" (UniqueName: \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930456    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.930644    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.931170    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.943749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.931417    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.946141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.946068    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.971336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.971297    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:09.974293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:09.974210    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:10.011725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:10.011650    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22vh\" (UniqueName: \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\") pod \"distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") " pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:10.059816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:10.059781    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:10.588005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:10.587964    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:21:10.589472 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:21:10.589443    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda09ada45_cd1f_4ca8_a326_373f37e106a2.slice/crio-eb8b03b63382dd697b55d19b93c30d80e2ab85b75ce3534dd954547f035a932f WatchSource:0}: Error finding container eb8b03b63382dd697b55d19b93c30d80e2ab85b75ce3534dd954547f035a932f: Status 404 returned error can't find the container with id eb8b03b63382dd697b55d19b93c30d80e2ab85b75ce3534dd954547f035a932f
Oct 31 16:21:10.973902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:10.973810    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" event={"ID":"a09ada45-cd1f-4ca8-a326-373f37e106a2","Type":"ContainerStarted","Data":"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"}
Oct 31 16:21:10.973902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:10.973867    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" event={"ID":"a09ada45-cd1f-4ca8-a326-373f37e106a2","Type":"ContainerStarted","Data":"eb8b03b63382dd697b55d19b93c30d80e2ab85b75ce3534dd954547f035a932f"}
Oct 31 16:21:11.060772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:11.060739    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:11.083390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:11.083333    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podStartSLOduration=2.083317854 podStartE2EDuration="2.083317854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:11.080043617 +0000 UTC m=+1116.141618802" watchObservedRunningTime="2026-10-31 16:21:11.083317854 +0000 UTC m=+1116.144893040"
Oct 31 16:21:20.060231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.060142    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:20.073136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.073109    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:20.387779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.387690    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:21:20.404486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.403168    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.419734    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.419816    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.419879    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.419911    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.419949    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.420085    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.420117    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.420143    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.421880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.420240    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2sw\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.491766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.491727    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:21:20.521366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521328    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521399    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521430    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521463    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521543    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521571    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521595    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521689    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2sw\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.521854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521795    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.522061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521921    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.522061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.521963    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.522061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.522000    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.522245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.522225    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.522684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.522658    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.524637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.524583    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.524637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.524585    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.574390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.574353    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.574547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.574459    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2sw\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:20.737250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:20.737150    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:21.057369 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:21.057337    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:21.154992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:21.154957    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:21:21.155724 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:21:21.155697    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc1ac91_b961_4fd1_b498_3c774e480a17.slice/crio-4f65e3456405c8355485b59caf56c112a493e00ac91cfdd8d687eee4add8c022 WatchSource:0}: Error finding container 4f65e3456405c8355485b59caf56c112a493e00ac91cfdd8d687eee4add8c022: Status 404 returned error can't find the container with id 4f65e3456405c8355485b59caf56c112a493e00ac91cfdd8d687eee4add8c022
Oct 31 16:21:21.158255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:21.158221    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:21.158337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:21.158285    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:21.158337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:21.158313    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:21:22.068858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.068809    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" event={"ID":"dfc1ac91-b961-4fd1-b498-3c774e480a17","Type":"ContainerStarted","Data":"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"}
Oct 31 16:21:22.069071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.068867    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" event={"ID":"dfc1ac91-b961-4fd1-b498-3c774e480a17","Type":"ContainerStarted","Data":"4f65e3456405c8355485b59caf56c112a493e00ac91cfdd8d687eee4add8c022"}
Oct 31 16:21:22.182515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.182459    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podStartSLOduration=2.18244352 podStartE2EDuration="2.18244352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:22.179595902 +0000 UTC m=+1127.241171089" watchObservedRunningTime="2026-10-31 16:21:22.18244352 +0000 UTC m=+1127.244018707"
Oct 31 16:21:22.746644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.745991    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:22.763643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.758731    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:22.763643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:22.758790    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:23.753649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:23.748058    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:23.753649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:23.748124    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:24.750642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:24.750046    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:24.750642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:24.750121    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:25.740638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:25.740045    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:25.740638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:25.740127    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:26.356456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.356405    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:21:26.361802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.361770    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.402472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.402434    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:21:26.455837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.445579    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"distinct-2-w0-ac1273-kserve-self-signed-certs\""
Oct 31 16:21:26.455837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.445720    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"openshift-service-ca.crt\""
Oct 31 16:21:26.464728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.457119    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"default-dockercfg-4f4mn\""
Oct 31 16:21:26.464728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.457204    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"kube-root-ca.crt\""
Oct 31 16:21:26.536694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.525255    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5zs\" (UniqueName: \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.593031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.592990    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.593423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.593395    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.593579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.593564    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.593755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.593733    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.594026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.594008    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.695729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695638    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.695729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695712    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.695967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695750    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.695967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695790    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.695967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695890    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.696121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.695978    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5zs\" (UniqueName: \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.696121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.696116    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.696232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.696171    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.696428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.696406    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.698120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.698104    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.698470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.698452    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.747643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.741031    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:26.747643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.741108    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:26.759593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.759514    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5zs\" (UniqueName: \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\") pod \"distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:26.978732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:26.978610    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:27.374938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:27.374907    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:21:27.377317 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:21:27.377280    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda908cd75_83cc_403d_998b_188faa247149.slice/crio-582b2e3f6c5e44157cf423199073cde99a627f0ee6b9ce1adf10f172b9c705c6 WatchSource:0}: Error finding container 582b2e3f6c5e44157cf423199073cde99a627f0ee6b9ce1adf10f172b9c705c6: Status 404 returned error can't find the container with id 582b2e3f6c5e44157cf423199073cde99a627f0ee6b9ce1adf10f172b9c705c6
Oct 31 16:21:27.739057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:27.738964    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 16:21:27.739057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:27.739031    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 16:21:28.122287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.122245    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" event={"ID":"a908cd75-83cc-403d-998b-188faa247149","Type":"ContainerStarted","Data":"985be2299149b2afadbaec9fa35749ca8ddf1edccf20bbc1d9410f40b982ac29"}
Oct 31 16:21:28.122770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.122298    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" event={"ID":"a908cd75-83cc-403d-998b-188faa247149","Type":"ContainerStarted","Data":"582b2e3f6c5e44157cf423199073cde99a627f0ee6b9ce1adf10f172b9c705c6"}
Oct 31 16:21:28.228473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.228411    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podStartSLOduration=2.228391519 podStartE2EDuration="2.228391519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:28.226542221 +0000 UTC m=+1133.288117406" watchObservedRunningTime="2026-10-31 16:21:28.228391519 +0000 UTC m=+1133.289966707"
Oct 31 16:21:28.743830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.743792    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:28.744213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.744185    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:28.745049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.745026    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:21:28.978925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:28.978832    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:32.945223 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:21:32.945184    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:21:36.979350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:36.979307    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:36.993525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:36.993495    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:37.206292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:37.206259    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:40.060019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.059981    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:21:40.064549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.064525    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.134485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.134442    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.135200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.135174    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.135405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.135386    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.135720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.135697    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.135876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.135858    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjrz\" (UniqueName: \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.136122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.136106    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.166025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.165990    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"default-dockercfg-hnnx7\""
Oct 31 16:21:40.170646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.166384    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"distinct-2-w5-ef6d93-kserve-self-signed-certs\""
Oct 31 16:21:40.174637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.170921    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"openshift-service-ca.crt\""
Oct 31 16:21:40.183680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.175876    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"kube-root-ca.crt\""
Oct 31 16:21:40.217664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.217601    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:21:40.242732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.242634    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.243057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.243033    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjrz\" (UniqueName: \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.251832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.243282    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.252121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.252104    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.252347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.252332    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.252559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.252526    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.253257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.253238    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.253791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.253763    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.255696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.255332    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.261387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.261352    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.268118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.268048    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.506598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.488772    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjrz\" (UniqueName: \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\") pod \"distinct-2-w5-ef6d93-kserve-569677974b-brwd8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") " pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:40.684270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:40.683891    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:41.567234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:41.567191    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:21:42.102744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.101721    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.260642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.259774    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.449707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.431541    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" event={"ID":"941784d0-03b0-4dd7-9c5f-5d5ba03827e8","Type":"ContainerStarted","Data":"7b0697dce5cffd99564a75869d8054797e113ac8ebdccc2074373d5773406da8"}
Oct 31 16:21:42.662846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.662806    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:42.663504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.663470    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:42.726698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:42.726152    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.097695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.061416    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 16:21:43.115643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.105393    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.127645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.124804    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.261755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.261703    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.349854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.349771    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:21:43.350475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.350453    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:43.367041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.365785    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.532172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.532133    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:43.540782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.540720    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.729480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.729375    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:43.731392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:43.731366    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:21:44.165783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.157064    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:21:44.165783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.165596    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:44.165783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.165680    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.188668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.166958    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:21:44.188668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.167348    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:44.212924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.211273    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:44.212924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.211364    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.282901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.277009    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.576659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.575077    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:21:44.576659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.575152    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:21:44.607163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.607024    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:21:44.607163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.607090    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.737683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.734138    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:21:44.763259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.763228    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:21:44.794019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.793983    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" event={"ID":"941784d0-03b0-4dd7-9c5f-5d5ba03827e8","Type":"ContainerStarted","Data":"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"}
Oct 31 16:21:44.872028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:44.871945    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:21:45.078216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:45.078166    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podStartSLOduration=5.0781518 podStartE2EDuration="5.0781518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:45.075442355 +0000 UTC m=+1150.137017542" watchObservedRunningTime="2026-10-31 16:21:45.0781518 +0000 UTC m=+1150.139726985"
Oct 31 16:21:45.685194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:45.685160    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:50.686450 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:50.686411    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:50.699260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:50.699235    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:21:50.843379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:21:50.843345    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:22:02.945233 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:22:02.945196    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:22:04.523715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.523677    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:22:04.528104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.528076    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.690725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.690605    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.690958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.690786    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.690958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.690812    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.690958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.690853    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.691112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.691009    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6cm\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.691147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.691123    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.691244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.691230    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.691301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.691253    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.691356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.691322    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.729748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.729278    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:22:04.799571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.799525    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.799844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.799586    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.799844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.799657    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.799844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.799690    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6cm\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800201    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800299    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800399    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800440    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800476    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800574    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.800968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.800942    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.802356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.802054    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.813084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.804426    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.818641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.813329    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.867683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.841195    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.867683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.848420    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.906594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.903452    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.907833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.907763    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6cm\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:04.938584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.938537    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:22:04.944907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:04.944881    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.001706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.001671    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:22:05.107119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107028    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107095    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107188    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107291    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107323    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107352    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107440    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107480    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.107559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.107523    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.142608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.142572    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.208855    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.208941    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209015    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209093    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209173    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209223    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209318    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209350    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.209400    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.210405    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.212107    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.212378    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.213050    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.217597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.211754    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.221125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.221094    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.222411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.222386    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.263448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.263414    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.265345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.265318    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.265581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.265565    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:05.563796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.563765    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:22:05.565202 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:05.565169    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cdad85_dd92_4e37_9562_2acd458fc1d8.slice/crio-362de64bda35878486c4a825df83b37456dcd9762600279d7545d85aa267b307 WatchSource:0}: Error finding container 362de64bda35878486c4a825df83b37456dcd9762600279d7545d85aa267b307: Status 404 returned error can't find the container with id 362de64bda35878486c4a825df83b37456dcd9762600279d7545d85aa267b307
Oct 31 16:22:05.568003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.567966    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.568114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.568048    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.568114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.568092    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.670677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.670645    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:22:05.673019 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:05.672505    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b607a_4354_403f_a966_fb91dc414bf7.slice/crio-47410adf07ced1fadfe757ce3a271335f75b4116ff72bbff8da8839455b561c3 WatchSource:0}: Error finding container 47410adf07ced1fadfe757ce3a271335f75b4116ff72bbff8da8839455b561c3: Status 404 returned error can't find the container with id 47410adf07ced1fadfe757ce3a271335f75b4116ff72bbff8da8839455b561c3
Oct 31 16:22:05.675553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.675515    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.675700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.675582    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.675700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.675629    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:05.709329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.709295    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:22:05.718754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.718725    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.779593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.779559    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:22:05.817466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817382    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817422    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817498    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817524    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817540    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817558    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sddg\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817581    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817657    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.817820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.817773    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.919657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919597    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.919877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919678    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.919877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919791    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.919877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919833    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.919877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919861    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919890    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sddg\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919944    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.919986    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920002    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920055    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920367 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920195    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920367 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920263    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920409    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.920699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.920654    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.922287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.922259    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.922819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.922795    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.974175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.974149    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:05.974788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.974745    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" event={"ID":"393b607a-4354-403f-a966-fb91dc414bf7","Type":"ContainerStarted","Data":"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"}
Oct 31 16:22:05.975068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.975041    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" event={"ID":"393b607a-4354-403f-a966-fb91dc414bf7","Type":"ContainerStarted","Data":"47410adf07ced1fadfe757ce3a271335f75b4116ff72bbff8da8839455b561c3"}
Oct 31 16:22:05.978225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.978193    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" event={"ID":"72cdad85-dd92-4e37-9562-2acd458fc1d8","Type":"ContainerStarted","Data":"0f88b900f3de183f042cfee56b489317b3f4506f4935bbeba4c63eea840697a2"}
Oct 31 16:22:05.978432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.978266    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" event={"ID":"72cdad85-dd92-4e37-9562-2acd458fc1d8","Type":"ContainerStarted","Data":"362de64bda35878486c4a825df83b37456dcd9762600279d7545d85aa267b307"}
Oct 31 16:22:05.987165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:05.987132    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sddg\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:06.049419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.049381    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:06.087489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.087382    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podStartSLOduration=2.087361537 podStartE2EDuration="2.087361537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.084218157 +0000 UTC m=+1171.145793344" watchObservedRunningTime="2026-10-31 16:22:06.087361537 +0000 UTC m=+1171.148936722"
Oct 31 16:22:06.143982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.143827    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:06.145726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.145672    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:06.146135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.146011    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:06.210517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.206394    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podStartSLOduration=2.20637338 podStartE2EDuration="2.20637338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.202975298 +0000 UTC m=+1171.264550495" watchObservedRunningTime="2026-10-31 16:22:06.20637338 +0000 UTC m=+1171.267948567"
Oct 31 16:22:06.266060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.266022    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:06.267707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.267674    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:06.267837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.267728    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:06.436906 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:06.436873    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcf44fe_5eeb_45e6_9806_7429c76cb2e1.slice/crio-5a8e8cb0839a76da1526dce3695889d410ef53a3322b056d0b8b98af28dafcbb WatchSource:0}: Error finding container 5a8e8cb0839a76da1526dce3695889d410ef53a3322b056d0b8b98af28dafcbb: Status 404 returned error can't find the container with id 5a8e8cb0839a76da1526dce3695889d410ef53a3322b056d0b8b98af28dafcbb
Oct 31 16:22:06.439671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.439598    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.439785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.439723    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.439785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.439768    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:06.444229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:06.444203    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:22:07.012783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.012634    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" event={"ID":"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1","Type":"ContainerStarted","Data":"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"}
Oct 31 16:22:07.012783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.012701    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" event={"ID":"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1","Type":"ContainerStarted","Data":"5a8e8cb0839a76da1526dce3695889d410ef53a3322b056d0b8b98af28dafcbb"}
Oct 31 16:22:07.059130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.058662    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.059130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.058722    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:07.069119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.069081    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:07.131196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.131110    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podStartSLOduration=2.131090336 podStartE2EDuration="2.131090336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:07.128935238 +0000 UTC m=+1172.190510425" watchObservedRunningTime="2026-10-31 16:22:07.131090336 +0000 UTC m=+1172.192665523"
Oct 31 16:22:07.153772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.153734    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.154040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.154011    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:07.269842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.269741    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:07.270103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:07.270074    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:08.051194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.051158    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.051762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.051731    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:08.145157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.145111    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.145443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.145413    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:08.267247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.267209    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:08.267526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:08.267481    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:09.067642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.067395    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.067642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.067454    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:09.153814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.153777    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.154063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.154031    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:09.268127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.268003    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:09.268127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:09.268065    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:10.202337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.175253    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.202337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.175316    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:10.352085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.291067    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.352085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.291158    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:10.352085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.295738    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:10.352085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:10.295792    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:11.099645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.098004    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.099645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.098081    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:11.131549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.131436    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.143776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.143657    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.143776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.143716    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:11.188923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.183383    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:22:11.318060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.317932    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:11.326762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.326720    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:11.355253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.331753    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.718920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.718809    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.778641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.776725    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:11.893648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:11.870135    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:12.068647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.067304    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.068647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.067375    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:12.219641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.219086    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.219641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.219163    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:12.321705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.319116    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:12.321705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.319177    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:12.412656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.387708    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:12.434651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.428703    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:22:12.592784 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:12.592682    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:22:13.081646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.058722    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.081970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.081929    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:13.145123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.145086    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.145457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.145416    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 16:22:13.267536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.267407    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 16:22:13.267536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:13.267473    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 16:22:14.052005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.051965    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:14.052607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.052574    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:14.326039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.325943    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:14.326647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.326596    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:14.328023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.327996    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:22:14.363427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.363392    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:14.365458 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.365432    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:14.385040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:14.385004    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:22:15.162641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:15.150418    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 16:22:15.162641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:15.150485    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 16:22:16.890847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:16.890791    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.092761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:17.092716    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:17.093063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:17.093030    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:17.578485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:17.578437    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:18.024998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:18.024861    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:18.045905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:18.045865    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:22:18.124706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:18.124669    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:22:18.146825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:18.146776    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:22:18.588264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:18.588233    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:22:19.187204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.187162    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:19.187663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.187223    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:19.190851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.190822    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:19.191142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.191119    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:19.191806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.191789    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:22:19.629967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.629927    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:22:19.636160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.636135    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.691340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.691303    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:22:19.825817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.825776    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.825817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.825820    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.825845    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826010    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826048    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jck66\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826142    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826182    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826251    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.826332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.826309    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928256    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928399    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928433    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928538    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928758    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928795    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928798    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.928945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928922    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.929027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.928995    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jck66\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.929122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.929106    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.929491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.929469    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.930167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.929780    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.930167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.929828    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.930296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.930116    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.932229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.932208    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.932664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.932644    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.941586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.941550    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:22:19.946996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.946975    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:19.981747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.981713    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:19.981913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:19.981858    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jck66\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:20.022817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.022779    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:22:20.131793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.131757    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.131988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.131942    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.131993    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132051    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132083    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132114    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132290    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132395    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.132444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.132430    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4sr\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.233598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.233511    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.233598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.233584    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4sr\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.233845    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234046    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234084    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234179    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234212    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234252    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234314    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234419    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234497    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234578    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.234856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234830    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.235049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.234955    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.236672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.236646    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.236779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.236752    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.253266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.253225    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:20.286148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.286117    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.286148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.286147    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4sr\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.574181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.574134    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:20.697141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.697095    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:22:20.709912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.708255    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:20.709912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.708338    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:20.709912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:20.708385    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:21.014167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.014072    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:22:21.024087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.024035    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:21.024208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.024128    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:21.024208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.024158    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:21.107822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.107784    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" event={"ID":"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74","Type":"ContainerStarted","Data":"01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a"}
Oct 31 16:22:21.108036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.107852    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" event={"ID":"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74","Type":"ContainerStarted","Data":"a72874b69c4b77db188bd8d4b530b174ad60db1a2d56412503b2875aaf8ece38"}
Oct 31 16:22:21.109935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.109904    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" event={"ID":"90b0f869-a6fa-4bb5-82fc-eccf8a08f759","Type":"ContainerStarted","Data":"47d5d7ad276c262fadd8bea255a0bb581212381cebf279903b41fedc00d229ae"}
Oct 31 16:22:21.110067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.109954    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" event={"ID":"90b0f869-a6fa-4bb5-82fc-eccf8a08f759","Type":"ContainerStarted","Data":"e1a1dcc733542ac49891010f0d853d59993d58b1c313a277ae20d2af91c3b113"}
Oct 31 16:22:21.216967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.216913    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podStartSLOduration=2.216896559 podStartE2EDuration="2.216896559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:21.213877241 +0000 UTC m=+1186.275452419" watchObservedRunningTime="2026-10-31 16:22:21.216896559 +0000 UTC m=+1186.278471744"
Oct 31 16:22:21.253579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.253546    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:21.254871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.254845    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:21.255005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.254892    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:21.323227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.323166    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podStartSLOduration=2.323147571 podStartE2EDuration="2.323147571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:21.319599503 +0000 UTC m=+1186.381174687" watchObservedRunningTime="2026-10-31 16:22:21.323147571 +0000 UTC m=+1186.384722757"
Oct 31 16:22:21.575161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:21.575066    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:22.257071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:22.257030    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:22.257718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:22.257686    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:22.579185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:22.579150    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:22.579466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:22.579437    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:23.262134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:23.262088    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:23.262795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:23.262756    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:23.575069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:23.575022    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:23.575293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:23.575108    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:24.255640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:24.255084    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:24.255640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:24.255146    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:24.589675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:24.586569    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:24.589675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:24.586649    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:25.262642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:25.262024    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:25.262642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:25.262085    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:25.631644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:25.624138    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:25.631644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:25.624223    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:26.258806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:26.258772    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:26.259091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:26.259061    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:26.582642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:26.582061    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:26.582642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:26.582122    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:27.253634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:27.253590    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 16:22:27.254037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:27.253665    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 16:22:27.575429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:27.575384    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 16:22:27.575692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:27.575464    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 16:22:28.332067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.332029    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:28.333647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.333597    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:28.335124 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.335108    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:22:28.659373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.659289    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:28.661933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.661906    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:28.662321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:28.662290    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:22:32.946046 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:22:32.945996    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:22:34.938034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:34.937986    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:22:34.942673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:34.942651    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.013685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.013653    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx5wl\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.013861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.013733    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.013861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.013801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.013982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.013929    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.014097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.014080    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.014153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.014141    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.014193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.014182    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.014244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.014211    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.014359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.014331    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.078177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.078145    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:22:35.116099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116061    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116136    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116200    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116244    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116277    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116394    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx5wl\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116484    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116540    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116686    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.116767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.116736    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.117139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.117104    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.117677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.117651    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.117821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.117788    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.117946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.117920    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.119409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.119390    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.120296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.120249    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.175634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.174903    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.185911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.185873    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx5wl\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.264532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.264445    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:35.757680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.757204    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:22:35.759156 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:35.759126    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4af8964_24c5_41e7_b001_b5e65fe721e3.slice/crio-78587a1a6622ae541f82e15d8370c55637d8242ab6564fae5457613469205afc WatchSource:0}: Error finding container 78587a1a6622ae541f82e15d8370c55637d8242ab6564fae5457613469205afc: Status 404 returned error can't find the container with id 78587a1a6622ae541f82e15d8370c55637d8242ab6564fae5457613469205afc
Oct 31 16:22:35.762073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.762039    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.762153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.762113    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.762153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.762148    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:35.942146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.942110    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:22:35.947049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.947022    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:35.962662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.962634    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:22:35.967564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:35.967541    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.006447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.006364    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:22:36.033000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.032963    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033022    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033122    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033192    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzgw\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033222    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033248    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033279    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033355    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033385    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033409    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033444    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlq6\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033486    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033537    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033653    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033720    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033762    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.033969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033808    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.033969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.033880    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.053086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.052894    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:22:36.136260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136222    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.136473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136302    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136344    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136384    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.136695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136483    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.136695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136574    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136610    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136663    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136731    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzgw\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.136919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136759    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.136919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136786    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136819    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.136919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136913    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.137166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136945    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.137166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.136971    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.137166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.137009    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlq6\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.137166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.137059    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.137166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.137112    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.137605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.137581    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.138909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.138884    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.139329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.139308    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.139457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.139320    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.139571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.139550    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.140057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.140033    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.141359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.140677    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.141359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.140915    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.141359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.141132    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.141939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.141915    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.142963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.142937    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.143815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.143793    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.147737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.147715    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.151637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.149202    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.191444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.191361    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.192872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.191817    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzgw\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.192872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.191940    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.193182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.193162    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlq6\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.263227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.263131    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:36.284949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.284909    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:36.313563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.313513    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" event={"ID":"d4af8964-24c5-41e7-b001-b5e65fe721e3","Type":"ContainerStarted","Data":"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"}
Oct 31 16:22:36.313786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.313581    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" event={"ID":"d4af8964-24c5-41e7-b001-b5e65fe721e3","Type":"ContainerStarted","Data":"78587a1a6622ae541f82e15d8370c55637d8242ab6564fae5457613469205afc"}
Oct 31 16:22:36.436647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.436501    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podStartSLOduration=2.436477107 podStartE2EDuration="2.436477107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:36.433005264 +0000 UTC m=+1201.494580451" watchObservedRunningTime="2026-10-31 16:22:36.436477107 +0000 UTC m=+1201.498052294"
Oct 31 16:22:36.690568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.690534    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:22:36.691786 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:36.691756    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5d3d3f_6625_4f72_9bec_7d8d1bb6127a.slice/crio-c479f255835d2300f2eccddf94261b4f993f29847bc8c739baa797cab4c5af39 WatchSource:0}: Error finding container c479f255835d2300f2eccddf94261b4f993f29847bc8c739baa797cab4c5af39: Status 404 returned error can't find the container with id c479f255835d2300f2eccddf94261b4f993f29847bc8c739baa797cab4c5af39
Oct 31 16:22:36.695035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.694986    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.695164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.695147    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.695233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.695219    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.766148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.750213    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:22:36.775154 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:22:36.775115    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf122eb5_87f2_409c_b18c_80b940adb05f.slice/crio-36efa292232f03f768dda9f5fbadb0e0a24182f7fb17265dfb922120b5660a69 WatchSource:0}: Error finding container 36efa292232f03f768dda9f5fbadb0e0a24182f7fb17265dfb922120b5660a69: Status 404 returned error can't find the container with id 36efa292232f03f768dda9f5fbadb0e0a24182f7fb17265dfb922120b5660a69
Oct 31 16:22:36.778051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.778012    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.778167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.778084    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:36.778167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:36.778122    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:22:37.277539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.277503    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:37.278339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.278314    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:37.278524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.278495    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:37.323387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.323351    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" event={"ID":"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a","Type":"ContainerStarted","Data":"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"}
Oct 31 16:22:37.323551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.323411    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" event={"ID":"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a","Type":"ContainerStarted","Data":"c479f255835d2300f2eccddf94261b4f993f29847bc8c739baa797cab4c5af39"}
Oct 31 16:22:37.326254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.326230    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" event={"ID":"af122eb5-87f2-409c-b18c-80b940adb05f","Type":"ContainerStarted","Data":"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"}
Oct 31 16:22:37.326427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.326414    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" event={"ID":"af122eb5-87f2-409c-b18c-80b940adb05f","Type":"ContainerStarted","Data":"36efa292232f03f768dda9f5fbadb0e0a24182f7fb17265dfb922120b5660a69"}
Oct 31 16:22:37.437645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.434768    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podStartSLOduration=2.434748912 podStartE2EDuration="2.434748912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.433404042 +0000 UTC m=+1202.494979227" watchObservedRunningTime="2026-10-31 16:22:37.434748912 +0000 UTC m=+1202.496324099"
Oct 31 16:22:37.628038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:37.627964    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podStartSLOduration=2.627944535 podStartE2EDuration="2.627944535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.625642643 +0000 UTC m=+1202.687217832" watchObservedRunningTime="2026-10-31 16:22:37.627944535 +0000 UTC m=+1202.689519724"
Oct 31 16:22:38.282657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.281128    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:38.282657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.281189    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:38.300644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.297746    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:38.300644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.298768    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:38.304630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.301743    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:38.304630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.301802    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:38.304630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.301892    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:38.304630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:38.301915    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:39.173646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.168692    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.278656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.268019    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:39.278656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.268083    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:39.278656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.268815    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:39.278656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.268857    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:39.289064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.289024    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:39.289759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.289704    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:39.367037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.366242    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.582655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.582249    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.596789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.596731    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.597923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.597886    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:39.771747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:39.771698    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.196641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.196309    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.221648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.197142    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:22:40.254348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.240876    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"]
Oct 31 16:22:40.275971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.275452    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:40.275971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.275516    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:40.300642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.288162    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:40.331473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.331332    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:40.331473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.296861    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:40.331473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.331428    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:40.528669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.528549    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50d2c9e7-6mpxk"]
Oct 31 16:22:40.589460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.589413    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.621646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.595937    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:22:40.749389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.623807    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.842684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.833176    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:22:40.842684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.788760    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:40.842684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:40.840212    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:22:41.459896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.459844    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:41.563871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.563832    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:41.573708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.567706    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:41.763699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.748558    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded"
Oct 31 16:22:41.815639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.790478    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.842644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.814685    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:22:41.894418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.894379    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:41.901744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.901701    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:41.939883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.939771    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:41.939883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.939834    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:41.978816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.978739    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.006724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.981422    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.079706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:41.997961    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.394701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.354268    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:42.394701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.386808    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:42.394701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.386857    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:42.443431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.427877    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:42.634641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.584161    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.646120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.646029    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:42.680878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.680793    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:42.681154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.669450    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a155e1-9701-42cf-8677-2f9df5818f8a" path="/var/lib/kubelet/pods/95a155e1-9701-42cf-8677-2f9df5818f8a/volumes"
Oct 31 16:22:42.681811 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:22:42.681787    4417 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.635s"
Oct 31 16:22:42.776930 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.776899    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:22:42.821677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.733969    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 16:22:42.822078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.822034    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:22:42.826743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.826702    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.879651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.846976    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:22:42.879651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.852089    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:42.879651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.855695    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:42.980274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:42.980000    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:22:43.008222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.008163    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.090017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.089634    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.109637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.098837    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.418851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.418814    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:43.419115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.419085    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:43.423907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.423875    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:43.424156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.424126    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:43.554151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.554111    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:43.651838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.651802    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:43.659692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.651720    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.742012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.741908    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:43.742266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.742236    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.835699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.832363    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:43.842643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.836925    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:22:43.983944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:43.983898    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.050318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.050276    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.094640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.084722    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:44.094640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.084789    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.139640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.137532    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.227253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.227213    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:44.246768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.246718    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.349966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.343862    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:44.349966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.343951    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:44.470870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.470805    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:44.472050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.472003    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:44.508568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.508527    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:44.509026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.508984    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:44.715697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.714599    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 31 16:22:44.727990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.727940    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.793455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.793414    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:44.798756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.798706    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.875643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.868999    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:44.875643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.869070    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.875643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.869166    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:44.878826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.878781    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:22:44.992709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:44.984453    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:45.123673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.061035    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:45.196639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.183892    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:45.312729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.277686    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:45.313070 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.313027    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:45.313340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.313317    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:45.313469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.313449    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:45.313899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.313875    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:45.314060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.314039    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:45.971647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:45.967036    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:22:46.022291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.022244    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:22:46.023899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.023862    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:22:46.024155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.024128    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:46.172754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.082336    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:22:46.193365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.193221    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:46.217719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.217673    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:22:46.479184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.478465    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:46.479184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.478530    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:46.661694 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:22:46.660674    4417 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.475s"
Oct 31 16:22:46.687643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.662821    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:46.687643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.662884    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:46.687643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.664583    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:22:46.767916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.751792    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:46.767916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.751894    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:46.972817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:46.972132    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.031847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.031740    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.186647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.182741    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.223763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.223711    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.249773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.246297    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.276779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.276741    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:47.277058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.277024    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:47.294950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.294903    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:47.295219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.295174    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:47.295450 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.295431    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:47.295685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.295659    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:47.749774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.749668    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:47.798868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:47.798821    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.013852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.013717    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.036751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.023750    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 16:22:48.057642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.035080    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.224641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.224198    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.230639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.224962    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.263745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.263693    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:48.264454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.264374    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:48.264454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.264428    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:48.268637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.264689    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:22:48.268637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.266102    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:48.268637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.266137    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:48.295814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.295754    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:48.297897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.297838    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:48.930995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:48.930964    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:22:49.066696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.061691    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:22:49.225704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.224974    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:49.225704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.225391    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:22:49.271645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.264860    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:22:49.280912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.274742    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:49.280912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.274804    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:49.280912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.275583    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 16:22:49.280912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.275649    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 16:22:49.317644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.316965    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:49.317644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.317021    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:49.520753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.520654    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:22:49.526597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.525694    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:22:49.622632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.622410    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:22:49.622991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:49.622968    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:22:50.263675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.263635    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 16:22:50.264272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.264238    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 16:22:50.265800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.265761    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:50.286663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.286500    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 16:22:50.286663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.286580    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 16:22:50.910789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.910747    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:50.912284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:50.912256    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:22:51.264280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.264183    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:51.285669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.285608    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:51.691522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.691482    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:51.693000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.692974    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:22:51.831608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.831569    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:51.834491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:51.834462    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:22:58.045791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:58.045753    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"]
Oct 31 16:22:58.090661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:58.090608    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-505ac380-bqxhn"]
Oct 31 16:22:59.052252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:22:59.052215    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e01794a-9e0d-42d0-9462-4d30fdc4e4ba" path="/var/lib/kubelet/pods/8e01794a-9e0d-42d0-9462-4d30fdc4e4ba/volumes"
Oct 31 16:23:02.947389 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:02.947347    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:23:06.125119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.125076    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:23:06.125659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.125300    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy" containerID="cri-o://cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37" gracePeriod=30
Oct 31 16:23:06.170891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.170856    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:23:06.171120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.171070    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy" containerID="cri-o://43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4" gracePeriod=30
Oct 31 16:23:06.174842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.174810    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:23:06.175151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:06.175105    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy" containerID="cri-o://31db2d59016bfc18e5e15e960d78afac6f21b2a1033d07f76d88064c21ff2aaf" gracePeriod=30
Oct 31 16:23:11.054108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.054075    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"]
Oct 31 16:23:11.089312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.089273    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2bb2b5-7xbv2"]
Oct 31 16:23:11.520121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.520088    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:23:11.523954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.523915    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:23:11.551963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.551934    4417 generic.go:356] "Generic (PLEG): container finished" podID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerID="43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4" exitCode=0
Oct 31 16:23:11.552104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.551993    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"
Oct 31 16:23:11.552104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.552008    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" event={"ID":"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca","Type":"ContainerDied","Data":"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"}
Oct 31 16:23:11.552104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.552070    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx" event={"ID":"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca","Type":"ContainerDied","Data":"ae2935eb68da99a06333cb781c926fec9d6fcce9a8843e78babfcd18aacd1f72"}
Oct 31 16:23:11.552104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.552102    4417 scope.go:122] "RemoveContainer" containerID="43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"
Oct 31 16:23:11.553882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.553858    4417 generic.go:356] "Generic (PLEG): container finished" podID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerID="31db2d59016bfc18e5e15e960d78afac6f21b2a1033d07f76d88064c21ff2aaf" exitCode=0
Oct 31 16:23:11.554008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.553946    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" event={"ID":"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b","Type":"ContainerDied","Data":"31db2d59016bfc18e5e15e960d78afac6f21b2a1033d07f76d88064c21ff2aaf"}
Oct 31 16:23:11.555956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.555931    4417 generic.go:356] "Generic (PLEG): container finished" podID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerID="cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37" exitCode=0
Oct 31 16:23:11.556124 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.556100    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" event={"ID":"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111","Type":"ContainerDied","Data":"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"}
Oct 31 16:23:11.556188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.556155    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n" event={"ID":"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111","Type":"ContainerDied","Data":"808552477a7e0b68bb8e6aad859b0ca780288f74f24f18d745bbb1f71e5687c1"}
Oct 31 16:23:11.556244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.556236    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"
Oct 31 16:23:11.561200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.561178    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:23:11.565207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.565182    4417 scope.go:122] "RemoveContainer" containerID="43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"
Oct 31 16:23:11.565578 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:11.565553    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4\": container with ID starting with 43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4 not found: ID does not exist" containerID="43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"
Oct 31 16:23:11.565693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.565592    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4"} err="failed to get container status \"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4\": rpc error: code = NotFound desc = could not find container \"43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4\": container with ID starting with 43c9cc1ee070f563f41729ea3103501bc3268fe0527472729e955f64760ab6d4 not found: ID does not exist"
Oct 31 16:23:11.565693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.565630    4417 scope.go:122] "RemoveContainer" containerID="cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"
Oct 31 16:23:11.576216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.576189    4417 scope.go:122] "RemoveContainer" containerID="cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"
Oct 31 16:23:11.576588 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:11.576566    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37\": container with ID starting with cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37 not found: ID does not exist" containerID="cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"
Oct 31 16:23:11.576708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.576602    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37"} err="failed to get container status \"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37\": rpc error: code = NotFound desc = could not find container \"cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37\": container with ID starting with cd39a4bc6a630890e416509c8b837e514f3e496e5742482df44bca23379d2d37 not found: ID does not exist"
Oct 31 16:23:11.683363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683251    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.683870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683734    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.683870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683801    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.683870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683831    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683899    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.683960    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.684112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684026    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.684112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684073    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684100    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.684382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684176    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684206    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684239    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684389    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684422    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.684551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684504    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.684551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684534    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684603    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684667    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684711    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.684786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684743    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.685049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684799    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.685049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684836    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.685049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.684947    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.685208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.685059    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\") pod \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\" (UID: \"0323a3ec-c0ea-41dd-8038-4cd921cbf7ca\") "
Oct 31 16:23:11.685208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.685104    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\") pod \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\" (UID: \"6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111\") "
Oct 31 16:23:11.685208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.685142    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.685208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.685190    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\") pod \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\" (UID: \"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b\") "
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.686394    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.686715    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.686917    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.687130    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.687514    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.687547    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.687842    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.688045    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.688252    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.688350    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:11.688721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.688444    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689237    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689293    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689313    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689330    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689344    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689359    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689372    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689386    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689399    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.689998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689628    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.689998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.689846    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.690219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.690195    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.690440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.690419    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.690740 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.690716    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:23:11.691826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.691799    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:23:11.692010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.691993    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.692389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.692354    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.693244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.693220    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.697767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.697565    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:23:11.697767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.697574    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "kube-api-access-8r6g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.697767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.697578    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo" pod "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" (UID: "0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:23:11.697767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.697678    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg" pod "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" (UID: "0323a3ec-c0ea-41dd-8038-4cd921cbf7ca"). InnerVolumeSpecName "kube-api-access-c9ffg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.698051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.697822    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "kube-api-access-x6fxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:11.698774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.698754    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo" pod "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" (UID: "6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:23:11.791108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791075    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791105    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791117    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791126    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791157    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r6g6\" (UniqueName: \"kubernetes.io/projected/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-kube-api-access-8r6g6\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791167    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791176    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9ffg\" (UniqueName: \"kubernetes.io/projected/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-kube-api-access-c9ffg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791186    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791194    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791202    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791210    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791222    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791234    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791246    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791254    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fxv\" (UniqueName: \"kubernetes.io/projected/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-kube-api-access-x6fxv\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791263    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791272    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791280    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.791344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.791289    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:11.999411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:11.999286    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:23:12.012826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.005066    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jbh6n"]
Oct 31 16:23:12.102949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.102911    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:23:12.109077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.109035    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-h8jrx"]
Oct 31 16:23:12.566158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.566120    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj" event={"ID":"0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b","Type":"ContainerDied","Data":"1866d466ea1700779049c151ef2b3df835582dc49c5f0ed5525e5183bbc8568b"}
Oct 31 16:23:12.566373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.566191    4417 scope.go:122] "RemoveContainer" containerID="31db2d59016bfc18e5e15e960d78afac6f21b2a1033d07f76d88064c21ff2aaf"
Oct 31 16:23:12.566373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.566137    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"
Oct 31 16:23:12.721426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.721391    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:23:12.727090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:12.727053    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s2qtj"]
Oct 31 16:23:13.052791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.052753    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" path="/var/lib/kubelet/pods/0323a3ec-c0ea-41dd-8038-4cd921cbf7ca/volumes"
Oct 31 16:23:13.053510 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.053490    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" path="/var/lib/kubelet/pods/0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b/volumes"
Oct 31 16:23:13.054137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.054118    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9e4d2a-5e94-46be-9c92-3d982be6bc2a" path="/var/lib/kubelet/pods/3c9e4d2a-5e94-46be-9c92-3d982be6bc2a/volumes"
Oct 31 16:23:13.055217 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.055198    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" path="/var/lib/kubelet/pods/6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111/volumes"
Oct 31 16:23:13.224309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.224272    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-c8e62a"]
Oct 31 16:23:13.225108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225081    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy"
Oct 31 16:23:13.225108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225109    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy"
Oct 31 16:23:13.225290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225174    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy"
Oct 31 16:23:13.225290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225183    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy"
Oct 31 16:23:13.225290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225201    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy"
Oct 31 16:23:13.225290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225210    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy"
Oct 31 16:23:13.225424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225379    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea4a5a3-bd66-4bda-a91b-dc13a6d8f19b" containerName="istio-proxy"
Oct 31 16:23:13.225424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225396    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9d2fe5-7a8c-4c4d-95ee-ec10bc56b111" containerName="istio-proxy"
Oct 31 16:23:13.225424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.225410    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0323a3ec-c0ea-41dd-8038-4cd921cbf7ca" containerName="istio-proxy"
Oct 31 16:23:13.230112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.230090    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:13.282142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.282112    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-szvbq\""
Oct 31 16:23:13.286526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.286500    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-c8e62a"]
Oct 31 16:23:13.339332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.339221    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgvp\" (UniqueName: \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\") pod \"test-curl-7198-c8e62a\" (UID: \"e921eb2c-68f3-488a-8528-31a10733375d\") " pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:13.440360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.440322    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgvp\" (UniqueName: \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\") pod \"test-curl-7198-c8e62a\" (UID: \"e921eb2c-68f3-488a-8528-31a10733375d\") " pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:13.493167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.493132    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgvp\" (UniqueName: \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\") pod \"test-curl-7198-c8e62a\" (UID: \"e921eb2c-68f3-488a-8528-31a10733375d\") " pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:13.543708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.543677    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:13.919596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.919554    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-c8e62a"]
Oct 31 16:23:13.922223 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:13.922190    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode921eb2c_68f3_488a_8528_31a10733375d.slice/crio-8d0233660eec50da45dd07612c4e2a01f941d976422e2b14a7c46378a4d3d164 WatchSource:0}: Error finding container 8d0233660eec50da45dd07612c4e2a01f941d976422e2b14a7c46378a4d3d164: Status 404 returned error can't find the container with id 8d0233660eec50da45dd07612c4e2a01f941d976422e2b14a7c46378a4d3d164
Oct 31 16:23:13.924122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:13.924104    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:23:14.594687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:14.594642    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-c8e62a" event={"ID":"e921eb2c-68f3-488a-8528-31a10733375d","Type":"ContainerStarted","Data":"8d0233660eec50da45dd07612c4e2a01f941d976422e2b14a7c46378a4d3d164"}
Oct 31 16:23:16.999662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:16.988760    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.014643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.010560    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.102521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.098732    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:23:17.470644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.469790    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.833645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.833381    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.866656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.853181    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:17.984642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:17.983147    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 16:23:18.012637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.011741    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.012637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.012263    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:23:18.116692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.116476    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:23:18.117645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.117340    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:23:18.711776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.711720    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.867788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.867736    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.893508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.893474    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:23:18.936654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.917200    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:18.936654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:18.921412    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:23:19.087822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.028746    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.087822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.042542    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.149409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.134715    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.823652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.801061    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.950652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.939104    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:19.950652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:19.944891    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.070772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.070724    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.075923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.075824    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.188645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.164002    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.211561 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:20.211518    4417 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.078s"
Oct 31 16:23:20.215644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.211918    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:23:20.216004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.215979    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:23:20.532049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.531944    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:20.532402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.532364    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.819708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.807852    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:23:20.900780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.900738    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:20.901084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.901049    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:20.939955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.939906    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.005859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:20.964361    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.083313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.083204    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.084237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.084203    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.084448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.084429    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.201959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.201911    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:23:21.771640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.761720    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.851644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.850325    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:21.957893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.957752    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:21.957893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:21.957821    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.066849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.066806    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.067405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.067370    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.113060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.113011    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.351042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.113308    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:23:22.371842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.226103    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:22.505064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.505026    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:23:22.874032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:22.851978    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:23.090447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:23.090387    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:23.092064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:23.092026    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:23:23.383651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:23.366355    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:23.410720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:23.410676    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:23.855729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:23.852766    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:24.150717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.136409    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:23:24.151413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.150080    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:23:24.210578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.210548    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:23:24.378701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.374966    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:23:24.394818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.394776    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:23:24.466739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.429587    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:24.564184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.564140    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:24.727942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.727770    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:23:24.741934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.741906    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:23:24.834178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:24.834146    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:23:27.492018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:27.491976    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-c8e62a" event={"ID":"e921eb2c-68f3-488a-8528-31a10733375d","Type":"ContainerStarted","Data":"8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9"}
Oct 31 16:23:27.597241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:27.597185    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-c8e62a" podStartSLOduration=1.666878267 podStartE2EDuration="14.597170565s" totalImagesPullingTime="12.930292298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:27.594101684 +0000 UTC m=+1252.655676869" watchObservedRunningTime="2026-10-31 16:23:27.597170565 +0000 UTC m=+1252.658745751"
Oct 31 16:23:29.119401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:29.119334    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-c8e62a"]
Oct 31 16:23:29.122987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:29.122956    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-c8e62a"]
Oct 31 16:23:29.505576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:29.505478    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-c8e62a" podUID="e921eb2c-68f3-488a-8528-31a10733375d" containerName="test-curl-7198-c8e62a" containerID="cri-o://8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9" gracePeriod=2
Oct 31 16:23:29.552886 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:29.552835    4417 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-c8e62a\" is forbidden: User \"system:node:ip-10-0-140-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-84.us-west-2.compute.internal' and this object" podUID="e921eb2c-68f3-488a-8528-31a10733375d" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:31.778275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:31.778205    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:31.906880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:31.906840    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\" (UniqueName: \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\") pod \"e921eb2c-68f3-488a-8528-31a10733375d\" (UID: \"e921eb2c-68f3-488a-8528-31a10733375d\") "
Oct 31 16:23:31.908947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:31.908924    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp" pod "e921eb2c-68f3-488a-8528-31a10733375d" (UID: "e921eb2c-68f3-488a-8528-31a10733375d"). InnerVolumeSpecName "kube-api-access-qtgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:23:32.008412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.008378    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgvp\" (UniqueName: \"kubernetes.io/projected/e921eb2c-68f3-488a-8528-31a10733375d-kube-api-access-qtgvp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:23:32.528522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.528492    4417 generic.go:356] "Generic (PLEG): container finished" podID="e921eb2c-68f3-488a-8528-31a10733375d" containerID="8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9" exitCode=137
Oct 31 16:23:32.528731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.528586    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:32.528731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.528669    4417 scope.go:122] "RemoveContainer" containerID="8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9"
Oct 31 16:23:32.539176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.539158    4417 scope.go:122] "RemoveContainer" containerID="8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9"
Oct 31 16:23:32.539496 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:32.539471    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9\": container with ID starting with 8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9 not found: ID does not exist" containerID="8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9"
Oct 31 16:23:32.539581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:32.539506    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9"} err="failed to get container status \"8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9\": rpc error: code = NotFound desc = could not find container \"8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9\": container with ID starting with 8b92c3876226a752730f238258fe62c32f37bfe4a6c5a63e2aec717fa34ca5e9 not found: ID does not exist"
Oct 31 16:23:32.584681 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:32.584611    4417 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-c8e62a\" is forbidden: User \"system:node:ip-10-0-140-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-84.us-west-2.compute.internal' and this object" podUID="e921eb2c-68f3-488a-8528-31a10733375d" pod="opendatahub/test-curl-7198-c8e62a"
Oct 31 16:23:32.946208 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:23:32.946164    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:23:33.052874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:33.052839    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e921eb2c-68f3-488a-8528-31a10733375d" path="/var/lib/kubelet/pods/e921eb2c-68f3-488a-8528-31a10733375d/volumes"
Oct 31 16:23:34.985867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:34.985815    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:23:34.986599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:34.986577    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e921eb2c-68f3-488a-8528-31a10733375d" containerName="test-curl-7198-c8e62a"
Oct 31 16:23:34.986599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:34.986599    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e921eb2c-68f3-488a-8528-31a10733375d" containerName="test-curl-7198-c8e62a"
Oct 31 16:23:34.986746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:34.986735    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e921eb2c-68f3-488a-8528-31a10733375d" containerName="test-curl-7198-c8e62a"
Oct 31 16:23:34.991554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:34.991534    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051354    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051402    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pj2l\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051505    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051556    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051735    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051918    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.051992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.051975    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.052067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.052052    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.052113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.052080    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.110604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.110565    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:23:35.154168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154130    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154198    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154258    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154282    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154395    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154425    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pj2l\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154477    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154526    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154612    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154642    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.154842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.154680    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.155204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.155171    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.155372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.155348    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.157174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.157143    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.157444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.157406    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.157928 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.157899    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.218014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.217982    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.218201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.218036    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pj2l\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.307448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.307418    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:35.735777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.735720    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:23:35.738937 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:35.738908    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead36ef_9851_4d9a_a53a_9df732c6303a.slice/crio-a7d41888c97d32420e934cfe84e7aa12514ba7588dce82509a34fc0d037e11e5 WatchSource:0}: Error finding container a7d41888c97d32420e934cfe84e7aa12514ba7588dce82509a34fc0d037e11e5: Status 404 returned error can't find the container with id a7d41888c97d32420e934cfe84e7aa12514ba7588dce82509a34fc0d037e11e5
Oct 31 16:23:35.741776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.741741    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:35.741907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.741822    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:35.741907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:35.741852    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:36.585018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:36.584972    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" event={"ID":"dead36ef-9851-4d9a-a53a-9df732c6303a","Type":"ContainerStarted","Data":"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"}
Oct 31 16:23:36.585459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:36.585049    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" event={"ID":"dead36ef-9851-4d9a-a53a-9df732c6303a","Type":"ContainerStarted","Data":"a7d41888c97d32420e934cfe84e7aa12514ba7588dce82509a34fc0d037e11e5"}
Oct 31 16:23:36.715436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:36.715372    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podStartSLOduration=2.715350917 podStartE2EDuration="2.715350917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:36.711775141 +0000 UTC m=+1261.773350328" watchObservedRunningTime="2026-10-31 16:23:36.715350917 +0000 UTC m=+1261.776926104"
Oct 31 16:23:37.308540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:37.308498    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:37.310053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:37.310024    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:37.310177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:37.310079    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:38.308493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.308450    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:38.309048 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.308515    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:38.378706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.378656    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:23:38.384091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.384063    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.475639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.475162    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f27146d0-gw-openshift-default-dockercfg-7sbnd\""
Oct 31 16:23:38.479412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.479294    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:23:38.490588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490552    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.490817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490681    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.490817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490738    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.490817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490759    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.490817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szs62\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.491061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490885    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.491061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490944    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.491061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.490978    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.491061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.491018    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.619701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.593252    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.620099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620065    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.620287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620270    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.620457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620443    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.620661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620638    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.620824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620807    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.621003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.620985    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.621118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.621103    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.633030 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.632988    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szs62\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.670641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.611443    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.670916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.646992    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.671038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.647272    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.671134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.647964    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.671239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.629258    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.671325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.661366    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.699640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.699237    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.730209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.730147    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.773740 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.773696    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szs62\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\") pod \"e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") " pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:38.999700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:38.998996    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:39.324638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.315743    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:39.324638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.315814    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:39.669638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.666664    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:39.669638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.666773    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:39.669638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.666881    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:39.699396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.674205    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:23:39.804639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:39.798671    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerStarted","Data":"d6eaffd5302b20b19041e68df6861e47279711796b007e22a86915351a42fb92"}
Oct 31 16:23:40.094044 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.094003    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:23:40.100205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.100168    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.182707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.182111    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186249    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186322    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186492    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186531    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186568    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz9dh\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.186735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186640    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.187175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186813    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.187175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186850    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.187175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.186882    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.207067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.207034    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a08562e5-openshift-default-dockercfg-fm2z8\""
Oct 31 16:23:40.322737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.296575    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.323101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.323074    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.379975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.323250    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.454698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.403992    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:40.454698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.421287    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.454698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.453991    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.454698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.454240    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.455707    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.478824    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.479301    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.479740    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.479886    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.479820    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.480038    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz9dh\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.480567    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.484657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.480895    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.485546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.480204    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.490948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.486174    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.515636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.512077    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.575640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.574612    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz9dh\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\") pod \"e2e-ait-a08562e5-openshift-default-c86877577-gv4tf\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") " pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.727697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.724319    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:40.813942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.809843    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerStarted","Data":"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"}
Oct 31 16:23:40.923641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:40.922777    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podStartSLOduration=2.922758458 podStartE2EDuration="2.922758458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:40.922609749 +0000 UTC m=+1265.984184948" watchObservedRunningTime="2026-10-31 16:23:40.922758458 +0000 UTC m=+1265.984333645"
Oct 31 16:23:41.001429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.001345    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:41.012754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.010645    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:41.234845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.234784    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:23:41.300446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.300378    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:41.300777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.300749    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:41.301006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.300966    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:41.333690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.331496    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:41.333690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.331675    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:41.819712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.818775    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerStarted","Data":"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"}
Oct 31 16:23:41.819712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.818843    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerStarted","Data":"8de138f0ad31899a0328a5e3b472c65e6020e5463ec051a65f4a2260da15dc02"}
Oct 31 16:23:41.819712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.819097    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:41.821505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.821481    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:23:41.932285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:41.932226    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podStartSLOduration=1.9322045110000001 podStartE2EDuration="1.932204511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:41.928824538 +0000 UTC m=+1266.990399758" watchObservedRunningTime="2026-10-31 16:23:41.932204511 +0000 UTC m=+1266.993779700"
Oct 31 16:23:42.320715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.310010    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 16:23:42.320715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.310073    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 16:23:42.725140 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.725042    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:42.731722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.731693    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:42.827780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.827744    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:42.828892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:42.828857    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:23:43.748554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:43.748517    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:43.749113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:43.749080    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:43.750301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:43.750278    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:23:44.590275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.590241    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:23:44.595222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.595195    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.682879 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.682845    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-caa664e1-openshift-default-dockercfg-cbxhc\""
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.711579    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.711654    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tgt\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.715494    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.715665    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.727743    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.727801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.727908    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.727961    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.728103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.728035    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.842773    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.842877    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.842920    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.842977    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.843132    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.843157    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tgt\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.843203    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.843325    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.843393    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.853649    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:44.976778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:44.970933    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:23:45.097578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.096983    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.097578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.097362    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.098297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.098277    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.098505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.098489    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.098674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.098658    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.099162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.099144    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.143287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.126879    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tgt\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.169023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.168961    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.254907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.222376    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:45.819227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.819196    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:23:45.820957 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:45.820933    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4528110c_2fa8_496f_b184_eabf72d7165c.slice/crio-dc8cf138a443807a089ffd0b5344762779c37b0e80de93eb91cb07f3624e92a5 WatchSource:0}: Error finding container dc8cf138a443807a089ffd0b5344762779c37b0e80de93eb91cb07f3624e92a5: Status 404 returned error can't find the container with id dc8cf138a443807a089ffd0b5344762779c37b0e80de93eb91cb07f3624e92a5
Oct 31 16:23:45.823709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.823673    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:45.823807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.823745    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:45.823807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:45.823778    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:23:46.022319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.022282    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerStarted","Data":"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"}
Oct 31 16:23:46.022502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.022349    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerStarted","Data":"dc8cf138a443807a089ffd0b5344762779c37b0e80de93eb91cb07f3624e92a5"}
Oct 31 16:23:46.136307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.136191    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podStartSLOduration=2.136176237 podStartE2EDuration="2.136176237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:46.13257106 +0000 UTC m=+1271.194146261" watchObservedRunningTime="2026-10-31 16:23:46.136176237 +0000 UTC m=+1271.197751463"
Oct 31 16:23:46.227593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.227554    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:46.229191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.229162    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:23:46.229326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:46.229215    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:23:47.015644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.015264    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:23:47.035822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.035790    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.113642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.112603    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f27146d0\""
Oct 31 16:23:47.133368 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.133336    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f27146d0-dockercfg-bvnb6\""
Oct 31 16:23:47.588397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.588347    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6s6\" (UniqueName: \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.588792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.588763    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.593895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.593829    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:47.603649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.595848    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:23:47.605123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.604326    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:23:47.630664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.630630    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:47.632672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.632651    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:23:47.632854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.632837    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:23:47.632993 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.632484    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.691359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.691321    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng6q\" (UniqueName: \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.691552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.691524    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6s6\" (UniqueName: \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.691611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.691595    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.691728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.691711    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.692501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.692480    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.744354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.744317    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6s6\" (UniqueName: \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\") pod \"payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.793424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.793383    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.793603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.793563    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng6q\" (UniqueName: \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.794153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.794134    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.809037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.809006    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:47.846958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.846876    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng6q\" (UniqueName: \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\") pod \"payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") " pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:47.959266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:47.959190    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:48.221119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.221078    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:23:48.223438 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:48.223408    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4109246_58f0_4de7_9b30_e269a2109ac9.slice/crio-10694f8ffb5969227aa456a7fd29265a6124f5d35f357553de5181ee1ecc7347 WatchSource:0}: Error finding container 10694f8ffb5969227aa456a7fd29265a6124f5d35f357553de5181ee1ecc7347: Status 404 returned error can't find the container with id 10694f8ffb5969227aa456a7fd29265a6124f5d35f357553de5181ee1ecc7347
Oct 31 16:23:48.376954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.376905    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:23:48.382127 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:48.382097    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a803ee_edd8_4f38_a17e_69d93fdd32a0.slice/crio-8e6725d7b29e28eb18efc9b3ad8e11993cf8a6a4adfc9c730dacb20cea21d4b2 WatchSource:0}: Error finding container 8e6725d7b29e28eb18efc9b3ad8e11993cf8a6a4adfc9c730dacb20cea21d4b2: Status 404 returned error can't find the container with id 8e6725d7b29e28eb18efc9b3ad8e11993cf8a6a4adfc9c730dacb20cea21d4b2
Oct 31 16:23:48.626634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.626581    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" event={"ID":"b4109246-58f0-4de7-9b30-e269a2109ac9","Type":"ContainerStarted","Data":"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"}
Oct 31 16:23:48.626819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.626657    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" event={"ID":"b4109246-58f0-4de7-9b30-e269a2109ac9","Type":"ContainerStarted","Data":"10694f8ffb5969227aa456a7fd29265a6124f5d35f357553de5181ee1ecc7347"}
Oct 31 16:23:48.626819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.626688    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:23:48.628407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.628370    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" event={"ID":"e0a803ee-edd8-4f38-a17e-69d93fdd32a0","Type":"ContainerStarted","Data":"1562925f3c3c577750538846903451806d436fdb10c7be3a304196aff6f650cd"}
Oct 31 16:23:48.628537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.628434    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" event={"ID":"e0a803ee-edd8-4f38-a17e-69d93fdd32a0","Type":"ContainerStarted","Data":"8e6725d7b29e28eb18efc9b3ad8e11993cf8a6a4adfc9c730dacb20cea21d4b2"}
Oct 31 16:23:48.628647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.628586    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:48.733353 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.733302    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" podStartSLOduration=2.73328698 podStartE2EDuration="2.73328698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:48.730363396 +0000 UTC m=+1273.791938586" watchObservedRunningTime="2026-10-31 16:23:48.73328698 +0000 UTC m=+1273.794862208"
Oct 31 16:23:48.838048 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:48.837977    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" podStartSLOduration=1.8379596459999998 podStartE2EDuration="1.837959646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:48.83578583 +0000 UTC m=+1273.897361015" watchObservedRunningTime="2026-10-31 16:23:48.837959646 +0000 UTC m=+1273.899534833"
Oct 31 16:23:49.770730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:49.768647    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:23:49.818822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:49.818780    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:23:49.819000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:49.818921    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:49.900639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:49.897470    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:49.900639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:49.897601    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjdb\" (UniqueName: \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.014641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.011166    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a08562e5\""
Oct 31 16:23:50.107243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.102032    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a08562e5-dockercfg-w22xf\""
Oct 31 16:23:50.137341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.132282    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.137341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.132448    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjdb\" (UniqueName: \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.138444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.138411    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.191514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.191469    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjdb\" (UniqueName: \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\") pod \"payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") " pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.457167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.457066    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:50.845129 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:50.845088    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:23:50.847471 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:23:50.847439    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fb9347_7d38_4da9_b135_643ebe867ef9.slice/crio-22f33b1c61f117038118d0969b9039940cc39499ba5d83540868b9d3aaa6ff37 WatchSource:0}: Error finding container 22f33b1c61f117038118d0969b9039940cc39499ba5d83540868b9d3aaa6ff37: Status 404 returned error can't find the container with id 22f33b1c61f117038118d0969b9039940cc39499ba5d83540868b9d3aaa6ff37
Oct 31 16:23:51.781026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:51.780983    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" event={"ID":"36fb9347-7d38-4da9-b135-643ebe867ef9","Type":"ContainerStarted","Data":"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"}
Oct 31 16:23:51.781288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:51.781048    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" event={"ID":"36fb9347-7d38-4da9-b135-643ebe867ef9","Type":"ContainerStarted","Data":"22f33b1c61f117038118d0969b9039940cc39499ba5d83540868b9d3aaa6ff37"}
Oct 31 16:23:51.781288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:51.781190    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:23:51.890065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:51.890009    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" podStartSLOduration=2.889993315 podStartE2EDuration="2.889993315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:51.887071154 +0000 UTC m=+1276.948646340" watchObservedRunningTime="2026-10-31 16:23:51.889993315 +0000 UTC m=+1276.951568542"
Oct 31 16:23:56.241324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.241293    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:23:56.257977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.247564    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.358652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.358293    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-caa664e1-dockercfg-tbsnv\""
Oct 31 16:23:56.358652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.358450    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-caa664e1\""
Oct 31 16:23:56.434651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.382205    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:23:56.434651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.432932    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.434651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.433009    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszfq\" (UniqueName: \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.461417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.461373    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:23:56.470804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.468164    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.606641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.597658    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.606641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.597806    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszfq\" (UniqueName: \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.606641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.597919    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.606641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.601120    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9sd\" (UniqueName: \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.652642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.652538    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:56.745674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.737917    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.745674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.738069    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9sd\" (UniqueName: \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.745674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.739496    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:56.764742 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.764694    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:23:56.876723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.857343    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9sd\" (UniqueName: \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\") pod \"payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") " pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:57.028678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:56.929205    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszfq\" (UniqueName: \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\") pod \"payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:57.028678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:57.011801    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:23:57.055324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:57.051236    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:23:57.449568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:57.435693    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:57.492642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:57.490863    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:23:57.912644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:57.907053    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.021707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.021041    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 16:23:58.310645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.301099    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:23:58.492933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.492871    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.502254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.502204    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:23:58.931653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.930785    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:58.959658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:58.932677    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:23:59.065708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.050168    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.164642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.081713    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.164642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.124735    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.164642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.124794    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.164642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.125075    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:23:59.340896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.304458    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.400700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.399233    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.400700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.399305    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.447640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.446943    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.447640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.447049    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.469676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.448087    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:23:59.516815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.516767    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.770706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.753806    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.770706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.753877    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.839642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.829826    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:23:59.853919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.853575    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:59.854246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.854152    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.938782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.938719    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:59.991643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:23:59.952834    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:24:00.091643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.090074    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.091643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.090433    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:24:00.134708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.128466    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.134708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.128825    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.134708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.129432    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:24:00.134708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.129488    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:24:00.402728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.401667    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:24:00.458825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.458779    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:00.459115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.459082    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.467643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.465947    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:00.467643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.466001    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.556767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.556724    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:00.557397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.557360    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.564216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.564163    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:00.947258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:00.947204    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.140997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.108116    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.140997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.111708    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded"
Oct 31 16:24:01.140997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.140307    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:24:01.140997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.140023    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.414826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.414770    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:01.574810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:01.574766    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.112643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.109753    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.112643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.110453    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.112643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.111450    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.112992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.112811    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-f98mh" podUID="71f2171c-108c-4ae4-aad3-d0bc9cf1cd61" containerName="main" probeResult="failure" output="Get \"https://10.133.0.27:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.294642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.276005    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.348686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.277901    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.446718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.424859    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:02.446718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.424975    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:02.493775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.493705    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:24:02.595645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:02.592170    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.134696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.117285    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 16:24:03.134696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.117881    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.134696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.117990    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.303188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.303043    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.419002 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.418884    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.553651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.551359    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:24:03.632645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.629017    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:03.650792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.650736    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.763063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.732761    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:24:03.763063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.734119    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:03.860756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:03.860688    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:24:04.109654 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:24:04.103838    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:24:04.211667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.206108    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:04.211667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.206364    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:24:04.269687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.269290    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:24:04.331835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.331788    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:04.427644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.384259    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" event={"ID":"47587759-3e3a-46a8-83da-d678b985404c","Type":"ContainerStarted","Data":"8696d50672aa5ffe8e071446e5ed2b2ff3f8c6d697f5c1b6ec850d3e34e0a96b"}
Oct 31 16:24:04.464772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.464716    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:04.623055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.623010    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:24:04.623417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.623386    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:04.752650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:04.738726    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:05.308643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.265333    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:24:05.308643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.290637    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:05.336850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.336702    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:24:05.337177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.337087    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:05.337177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.337130    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:05.661874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.546854    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:24:05.832645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.777452    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:05.865660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:05.838932    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:06.129668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.114721    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" event={"ID":"768478da-1d69-4f3b-b90a-2e027a284d71","Type":"ContainerStarted","Data":"965693f2f445dfdc95264d213fbf92cca5f4f31838ffa9d40deead6889aceef1"}
Oct 31 16:24:06.528702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.516734    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:24:06.528702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.516958    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:06.528702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.518533    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:06.663773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.663718    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:24:06.779828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.779718    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:24:06.848176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:06.848116    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.286644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.266714    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" event={"ID":"768478da-1d69-4f3b-b90a-2e027a284d71","Type":"ContainerStarted","Data":"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"}
Oct 31 16:24:07.286644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.267962    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:24:07.311646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.311427    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" event={"ID":"47587759-3e3a-46a8-83da-d678b985404c","Type":"ContainerStarted","Data":"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"}
Oct 31 16:24:07.337642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.312681    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:24:07.388646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.381642    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" podStartSLOduration=11.381605156 podStartE2EDuration="11.381605156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:07.380268156 +0000 UTC m=+1292.441843342" watchObservedRunningTime="2026-10-31 16:24:07.381605156 +0000 UTC m=+1292.443180343"
Oct 31 16:24:07.417760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.417703    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.495926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.495357    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" podStartSLOduration=11.495337071 podStartE2EDuration="11.495337071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:07.494972063 +0000 UTC m=+1292.556547251" watchObservedRunningTime="2026-10-31 16:24:07.495337071 +0000 UTC m=+1292.556912257"
Oct 31 16:24:07.563660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.559963    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:24:07.580723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.580676    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.594686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.585697    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.674878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.674831    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:24:07.781774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.780788    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.782110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.782084    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:07.894779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.894678    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.895409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.861989    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.929166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.925767    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:07.929166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.925835    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:07.981099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:07.978924    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 16:24:08.177698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.172334    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:24:08.179263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.179231    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:24:08.229281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.229251    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:24:08.234568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.233695    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:24:08.234568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.234274    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:24:08.235905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:08.235883    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:24:19.330607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:19.330531    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:24:19.331117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:19.330609    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:24:32.949695 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:24:32.949654    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:24:35.337947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:35.337911    4417 scope.go:122] "RemoveContainer" containerID="452ce4750ad374a0661157a98209e67550d294b2cd3d3105ff716a84ba00c557"
Oct 31 16:24:35.350106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:35.350080    4417 scope.go:122] "RemoveContainer" containerID="76675e4958bd166ca2819221769825abc0b90b7f382aa6e9728f91d7b105a71d"
Oct 31 16:24:35.359583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:35.359562    4417 scope.go:122] "RemoveContainer" containerID="ea9d6109cadc1cdd5c414b81837771cd4de9fcd18c705bbeff4f5ebe903b07a0"
Oct 31 16:24:39.137641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:39.136743    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.309823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:39.309757    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.437706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:39.429961    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.510641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:39.509764    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:39.567773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:39.567727    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:24:40.095642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.081746    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:24:40.281676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.275168    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.281676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.276634    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:24:40.501738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.499690    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.501738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.500635    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:24:40.629664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.547291    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.629664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.548494    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:24:40.629664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.572242    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:40.629664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.574285    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:24:40.771707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.769135    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:40.771707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:40.769214    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.087641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.082916    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.110650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.109683    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.454701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.355226    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:24:41.589242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.499854    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.656698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.552713    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.663435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.580065    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.665209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.580123    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:41.921828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:41.921770    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.025997    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.026065    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.026103    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.026282    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.026349    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:24:42.027846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:24:42.026387    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:25:02.949025 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:02.948983    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:25:07.193037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.192999    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:25:07.198218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.198194    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224301    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbp5\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224352    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224470    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224661    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224694    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224721    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224819    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224870    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.224943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.224926    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.326474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.326433    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.326699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.326589    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.326699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.326638    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.326699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.326678    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.326947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.326923    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327173    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327287    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327351    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327671    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327733    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327786    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.327958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327934    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbp5\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.328093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.327983    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.328343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.328319    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.329346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.329326    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.330069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.330051    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.380957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.380922    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.381110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.381048    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbp5\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sjzsr\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.392325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.392291    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:25:07.516633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.512672    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:07.939976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.939938    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:25:07.942327 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:25:07.942300    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95d0713_61d2_48be_9b5d_bb76bc0c4489.slice/crio-5487fbd3148fd1d57a7550f6e63706c91bb3355614d03eea43c7ddde85864aee WatchSource:0}: Error finding container 5487fbd3148fd1d57a7550f6e63706c91bb3355614d03eea43c7ddde85864aee: Status 404 returned error can't find the container with id 5487fbd3148fd1d57a7550f6e63706c91bb3355614d03eea43c7ddde85864aee
Oct 31 16:25:07.946318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.946279    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:07.946430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.946366    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:07.946430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:07.946412    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:08.877997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:08.877954    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" event={"ID":"b95d0713-61d2-48be-9b5d-bb76bc0c4489","Type":"ContainerStarted","Data":"14e4322f0f4db42a39a33fbdedf6464f79f9b57d4b07d1b051be8bcdd58e2eac"}
Oct 31 16:25:08.878398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:08.878057    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" event={"ID":"b95d0713-61d2-48be-9b5d-bb76bc0c4489","Type":"ContainerStarted","Data":"5487fbd3148fd1d57a7550f6e63706c91bb3355614d03eea43c7ddde85864aee"}
Oct 31 16:25:09.084653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:09.081772    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podStartSLOduration=2.081753009 podStartE2EDuration="2.081753009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:09.034295344 +0000 UTC m=+1354.095870531" watchObservedRunningTime="2026-10-31 16:25:09.081753009 +0000 UTC m=+1354.143328200"
Oct 31 16:25:09.517789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:09.517748    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:09.519731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:09.519703    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:09.519868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:09.519766    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:10.517753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:10.517706    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:10.518242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:10.517788    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:11.517775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:11.517736    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:11.518181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:11.517817    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:12.517755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:12.517704    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:12.518189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:12.517811    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:13.517577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:13.517538    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:13.517837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:13.517608    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:14.517999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:14.517954    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:14.518411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:14.518016    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:15.518246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:15.518208    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 31 16:25:15.518720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:15.518272    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 31 16:25:16.522609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:16.522570    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:16.523115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:16.523095    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:16.523922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:16.523900    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:25:22.259843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.259807    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:25:22.264509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.264481    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.581774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.581724    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.689963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.689920    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsmd\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.706734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.690359    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.709401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.708204    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.711057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.709453    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:25:22.711299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.710682    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.711582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.711566    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.711878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.711860    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.712102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.712083    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.712260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.712245    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.818292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.818233    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsmd\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.820863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.820824    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.821632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.821594    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.822181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.822163    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.822804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.822782    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.823019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823005    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.823456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823439    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.823933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823917    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.824768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.824749    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.827976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.822106    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.828136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823855    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.828227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823112    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.828338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.824648    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.828422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.823365    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.831576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.831551    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.843322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.843197    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.882256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.882183    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:22.890412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:22.890333    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsmd\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-nbw47\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:23.188699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:23.188318    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:24.088650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:24.073832    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:25:24.129564 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:25:24.129533    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8191271_030d_4170_b05e_ac9c285d8f4d.slice/crio-fc5de55d8049b775d82b2845a4dd2770dcd3d3ebd7fef8bf9c24d55aa61e89be WatchSource:0}: Error finding container fc5de55d8049b775d82b2845a4dd2770dcd3d3ebd7fef8bf9c24d55aa61e89be: Status 404 returned error can't find the container with id fc5de55d8049b775d82b2845a4dd2770dcd3d3ebd7fef8bf9c24d55aa61e89be
Oct 31 16:25:24.132594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:24.132554    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:24.132725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:24.132645    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:24.132725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:24.132675    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:25.231573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:25.231513    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" event={"ID":"a8191271-030d-4170-b05e-ac9c285d8f4d","Type":"ContainerStarted","Data":"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"}
Oct 31 16:25:25.232257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:25.232227    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" event={"ID":"a8191271-030d-4170-b05e-ac9c285d8f4d","Type":"ContainerStarted","Data":"fc5de55d8049b775d82b2845a4dd2770dcd3d3ebd7fef8bf9c24d55aa61e89be"}
Oct 31 16:25:25.398884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:25.388538    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podStartSLOduration=3.388517445 podStartE2EDuration="3.388517445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:25.387136802 +0000 UTC m=+1370.448711989" watchObservedRunningTime="2026-10-31 16:25:25.388517445 +0000 UTC m=+1370.450092633"
Oct 31 16:25:26.204658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:26.204611    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:26.219642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:26.218658    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:26.219642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:26.218726    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:26.672638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:26.671654    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:26.703696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:26.688720    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:27.056733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.056686    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:25:27.056932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.056904    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:25:27.064857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.057245    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:25:27.201051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.201012    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:27.201317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.201286    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:27.323909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.323812    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:25:27.410760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.410706    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:27.694262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.694152    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:27.695409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:27.695383    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:25:28.119753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.085082    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:28.119753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.101767    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:25:28.169360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.120100    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 31 16:25:28.169360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.120349    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:25:28.169360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.120841    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:25:28.169360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.121770    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:25:28.169360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.122354    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:25:28.193659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.193081    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:28.193659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.193162    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:28.465705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.399910    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:28.715639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.688757    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:28.715639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.698661    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:25:28.715639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.714120    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:28.897642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:28.885494    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:25:29.096813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.096764    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-m4nbt" podUID="c73e7d4e-3bee-4d9c-af5a-7f63e560806a" containerName="server" probeResult="failure" output="Get \"https://10.133.0.24:8443/healthz\": context deadline exceeded"
Oct 31 16:25:29.153382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.153252    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 31 16:25:29.179866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.179809    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p" podUID="cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Oct 31 16:25:29.190883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.190837    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:29.227863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.227829    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:29.370019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.367718    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:29.424760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.424673    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:25:29.445969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.445924    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:25:29.515589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.514942    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:25:29.518292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:29.518228    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:25:30.190902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.190805    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:30.190902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.190881    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:30.568249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.568209    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:25:30.574163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.574140    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680375    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680477    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680523    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680547    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680598    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680681    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5cff\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680749    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680843    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.680856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.680866    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.773464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.773431    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:25:30.782236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782201    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782312    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782366    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782389    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782446    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782497    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cff\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782568    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782684    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.782758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.782710    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.783096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.783054    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.783404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.783376    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.783404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.783391    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.783647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.783599    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.784293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.784271    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.786561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.786539    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.789473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.789439    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.853541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.853418    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cff\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.870338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.870267    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:30.889467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:30.889338    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:31.190829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.190727    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:31.190829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.190792    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:31.365666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.357007    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:25:31.401170 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:25:31.401137    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beaed5d_9e83_4a3c_9f90_f038b05ad5de.slice/crio-15f57ba6149c2bd11d0c1a1252d48c8873a4ca8815b6c2c3db1caa033490b33d WatchSource:0}: Error finding container 15f57ba6149c2bd11d0c1a1252d48c8873a4ca8815b6c2c3db1caa033490b33d: Status 404 returned error can't find the container with id 15f57ba6149c2bd11d0c1a1252d48c8873a4ca8815b6c2c3db1caa033490b33d
Oct 31 16:25:31.405013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.404975    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.405103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.405072    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.405154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.405110    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:31.951641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.932532    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" event={"ID":"3beaed5d-9e83-4a3c-9f90-f038b05ad5de","Type":"ContainerStarted","Data":"5c2dbe4553bab41feaa6004cc52620aa723938baca3c65950e2896d105c586a0"}
Oct 31 16:25:31.951641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:31.932608    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" event={"ID":"3beaed5d-9e83-4a3c-9f90-f038b05ad5de","Type":"ContainerStarted","Data":"15f57ba6149c2bd11d0c1a1252d48c8873a4ca8815b6c2c3db1caa033490b33d"}
Oct 31 16:25:32.191715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:32.190874    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:32.191715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:32.190936    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:32.901640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:32.901576    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:32.901881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:32.901760    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:32.901881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:32.901827    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:33.011284 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:33.006287    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:25:33.212699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.192557    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 16:25:33.212699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.192646    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 16:25:33.661106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.660733    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:25:33.661106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.660805    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:33.892162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.890415    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:33.892162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:33.890501    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:34.893808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:34.892441    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:34.893808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:34.892507    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:35.236695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.236183    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-nbw47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:25:35.236695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.236252    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:35.466638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.466435    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:35.477641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.468126    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:35.477641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.474873    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:25:35.677862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.677574    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podStartSLOduration=5.677552218 podStartE2EDuration="5.677552218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:32.060554151 +0000 UTC m=+1377.122129351" watchObservedRunningTime="2026-10-31 16:25:35.677552218 +0000 UTC m=+1380.739127404"
Oct 31 16:25:35.897801 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.890024    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:35.897801 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:35.890100    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:36.890348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:36.890313    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:37.327604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:36.890374    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:37.889632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:37.889581    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 16:25:37.889866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:37.889651    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 16:25:38.096974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.096941    4417 generic.go:356] "Generic (PLEG): container finished" podID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerID="ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2" exitCode=0
Oct 31 16:25:38.097171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.097013    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerDied","Data":"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"}
Oct 31 16:25:38.097529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.097511    4417 scope.go:122] "RemoveContainer" containerID="ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"
Oct 31 16:25:38.098817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.098799    4417 generic.go:356] "Generic (PLEG): container finished" podID="4528110c-2fa8-496f-b184-eabf72d7165c" containerID="afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8" exitCode=0
Oct 31 16:25:38.098881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.098869    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerDied","Data":"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"}
Oct 31 16:25:38.099137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.099112    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.099200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.099186    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.099242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.099226    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.099325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.099229    4417 scope.go:122] "RemoveContainer" containerID="afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"
Oct 31 16:25:38.101416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.101389    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.101506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.101471    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.101567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.101521    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.101948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.101926    4417 generic.go:356] "Generic (PLEG): container finished" podID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerID="89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6" exitCode=0
Oct 31 16:25:38.102044 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.102025    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerDied","Data":"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"}
Oct 31 16:25:38.102471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.102443    4417 scope.go:122] "RemoveContainer" containerID="89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"
Oct 31 16:25:38.104423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.104378    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.104676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.104462    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.104676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.104507    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:25:38.918042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.918012    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:38.931650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.926080    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:38.940247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:38.940216    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:25:39.133642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.133571    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerStarted","Data":"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"}
Oct 31 16:25:39.142649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.140710    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerStarted","Data":"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"}
Oct 31 16:25:39.142649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.140758    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerStarted","Data":"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"}
Oct 31 16:25:39.227920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.227830    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:25:39.228274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.228246    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:39.228422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.228300    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:39.727642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.727410    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:25:39.727642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.727577    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:39.727642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:39.727639    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:40.002007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.001408    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:25:40.002007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.001554    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:40.002007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.001658    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:40.278704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.274909    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:40.278704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.274970    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:40.730669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.725027    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:40.730669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:40.725102    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:41.032202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.023964    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:41.032202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.031720    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:41.234122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.234083    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:41.234409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.234376    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:41.764775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.729972    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:41.766195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.766150    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:41.825671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:41.819082    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:25:42.056639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.053220    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:42.056639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.053296    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:42.228123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.227635    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:42.228123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.227709    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:42.727123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.727079    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:42.727344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.727145    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:42.829817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:42.829737    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:25:43.029976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.026938    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:43.029976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.027008    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:43.228193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.228157    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:43.228840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.228780    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:43.329181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.329089    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:25:43.496698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.487834    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:25:44.184638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.981983    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:44.184638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:43.982063    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:44.184638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.095747    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:44.184638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.095918    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:44.317090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.232155    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:44.317090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.232231    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:44.571051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.570686    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:44.751642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.743772    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:44.751642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:44.743858    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:45.000884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.000787    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:45.001330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.001285    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:45.233843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.233806    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 16:25:45.234107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.234079    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 16:25:45.578661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.574770    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:45.672771 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.672710    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:25:45.727636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.725827    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:25:45.731595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.731563    4417 patch_prober.go:29] interesting pod/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 16:25:45.731860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:45.731833    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 16:25:46.001728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.000976    4417 patch_prober.go:29] interesting pod/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 16:25:46.001728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.001048    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 16:25:46.280413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.280322    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:25:46.281342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.281313    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:25:46.282864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.282843    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:25:46.730359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.730322    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:25:46.731492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:46.731469    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:25:47.004996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:47.004877    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:25:47.006272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:47.006254    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:25:48.330058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.329990    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" podUID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" containerName="payload-processing" containerID="cri-o://1562925f3c3c577750538846903451806d436fdb10c7be3a304196aff6f650cd" gracePeriod=25
Oct 31 16:25:48.488749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.488673    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" podUID="b4109246-58f0-4de7-9b30-e269a2109ac9" containerName="payload-pre-processing" containerID="cri-o://0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc" gracePeriod=25
Oct 31 16:25:48.547563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.547529    4417 generic.go:356] "Generic (PLEG): container finished" podID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" containerID="1562925f3c3c577750538846903451806d436fdb10c7be3a304196aff6f650cd" exitCode=0
Oct 31 16:25:48.547751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.547599    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" event={"ID":"e0a803ee-edd8-4f38-a17e-69d93fdd32a0","Type":"ContainerDied","Data":"1562925f3c3c577750538846903451806d436fdb10c7be3a304196aff6f650cd"}
Oct 31 16:25:48.625116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.625089    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:25:48.767918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.767891    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\") pod \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") "
Oct 31 16:25:48.768274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.768251    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\" (UniqueName: \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\") pod \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\" (UID: \"e0a803ee-edd8-4f38-a17e-69d93fdd32a0\") "
Oct 31 16:25:48.768421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.768376    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume" pod "e0a803ee-edd8-4f38-a17e-69d93fdd32a0" (UID: "e0a803ee-edd8-4f38-a17e-69d93fdd32a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:48.769056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.769014    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:48.771387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.771360    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q" pod "e0a803ee-edd8-4f38-a17e-69d93fdd32a0" (UID: "e0a803ee-edd8-4f38-a17e-69d93fdd32a0"). InnerVolumeSpecName "kube-api-access-wng6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:48.772280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.772258    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:25:48.853100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.852129    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:25:48.869733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.869701    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\") pod \"b4109246-58f0-4de7-9b30-e269a2109ac9\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") "
Oct 31 16:25:48.869920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.869812    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\" (UniqueName: \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\") pod \"b4109246-58f0-4de7-9b30-e269a2109ac9\" (UID: \"b4109246-58f0-4de7-9b30-e269a2109ac9\") "
Oct 31 16:25:48.870099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.870078    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume" pod "b4109246-58f0-4de7-9b30-e269a2109ac9" (UID: "b4109246-58f0-4de7-9b30-e269a2109ac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:48.870533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.870516    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4109246-58f0-4de7-9b30-e269a2109ac9-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:48.870571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.870537    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wng6q\" (UniqueName: \"kubernetes.io/projected/e0a803ee-edd8-4f38-a17e-69d93fdd32a0-kube-api-access-wng6q\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:48.871993 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.871968    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6" pod "b4109246-58f0-4de7-9b30-e269a2109ac9" (UID: "b4109246-58f0-4de7-9b30-e269a2109ac9"). InnerVolumeSpecName "kube-api-access-wh6s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:48.975907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:48.975869    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6s6\" (UniqueName: \"kubernetes.io/projected/b4109246-58f0-4de7-9b30-e269a2109ac9-kube-api-access-wh6s6\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:49.555918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.555883    4417 generic.go:356] "Generic (PLEG): container finished" podID="b4109246-58f0-4de7-9b30-e269a2109ac9" containerID="0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc" exitCode=0
Oct 31 16:25:49.556390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.555936    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"
Oct 31 16:25:49.556390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.555965    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" event={"ID":"b4109246-58f0-4de7-9b30-e269a2109ac9","Type":"ContainerDied","Data":"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"}
Oct 31 16:25:49.556390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.556023    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp" event={"ID":"b4109246-58f0-4de7-9b30-e269a2109ac9","Type":"ContainerDied","Data":"10694f8ffb5969227aa456a7fd29265a6124f5d35f357553de5181ee1ecc7347"}
Oct 31 16:25:49.556390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.556051    4417 scope.go:122] "RemoveContainer" containerID="0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"
Oct 31 16:25:49.557798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.557758    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"
Oct 31 16:25:49.557798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.557763    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk" event={"ID":"e0a803ee-edd8-4f38-a17e-69d93fdd32a0","Type":"ContainerDied","Data":"8e6725d7b29e28eb18efc9b3ad8e11993cf8a6a4adfc9c730dacb20cea21d4b2"}
Oct 31 16:25:49.566552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.566537    4417 scope.go:122] "RemoveContainer" containerID="0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"
Oct 31 16:25:49.566866 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:49.566846    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc\": container with ID starting with 0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc not found: ID does not exist" containerID="0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"
Oct 31 16:25:49.566935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.566873    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc"} err="failed to get container status \"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc\": rpc error: code = NotFound desc = could not find container \"0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc\": container with ID starting with 0fe3fa0268cb601cc2366aaeeb4f0c3de2e9971a5eb762ce41502e0f4e34d3fc not found: ID does not exist"
Oct 31 16:25:49.566935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.566898    4417 scope.go:122] "RemoveContainer" containerID="1562925f3c3c577750538846903451806d436fdb10c7be3a304196aff6f650cd"
Oct 31 16:25:49.709067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.708964    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:25:49.714781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.714748    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f27146d0-775cb499fc-8qhmk"]
Oct 31 16:25:49.814469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.814427    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:25:49.820347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:49.820317    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f27146d0-5976fb74f9-vxqsp"]
Oct 31 16:25:51.051513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:51.051480    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4109246-58f0-4de7-9b30-e269a2109ac9" path="/var/lib/kubelet/pods/b4109246-58f0-4de7-9b30-e269a2109ac9/volumes"
Oct 31 16:25:51.052135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:51.052058    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" path="/var/lib/kubelet/pods/e0a803ee-edd8-4f38-a17e-69d93fdd32a0/volumes"
Oct 31 16:25:51.173313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:51.173277    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:25:51.298652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:51.295950    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:25:53.853651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:53.853584    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" podUID="36fb9347-7d38-4da9-b135-643ebe867ef9" containerName="payload-processing" containerID="cri-o://2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60" gracePeriod=25
Oct 31 16:25:54.167663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.167633    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:25:54.274841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.274809    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\" (UniqueName: \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\") pod \"36fb9347-7d38-4da9-b135-643ebe867ef9\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") "
Oct 31 16:25:54.275057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.275036    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\") pod \"36fb9347-7d38-4da9-b135-643ebe867ef9\" (UID: \"36fb9347-7d38-4da9-b135-643ebe867ef9\") "
Oct 31 16:25:54.275419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.275388    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume" pod "36fb9347-7d38-4da9-b135-643ebe867ef9" (UID: "36fb9347-7d38-4da9-b135-643ebe867ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:54.275725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.275706    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fb9347-7d38-4da9-b135-643ebe867ef9-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:54.277419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.277389    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb" pod "36fb9347-7d38-4da9-b135-643ebe867ef9" (UID: "36fb9347-7d38-4da9-b135-643ebe867ef9"). InnerVolumeSpecName "kube-api-access-kgjdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:54.378152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.378115    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjdb\" (UniqueName: \"kubernetes.io/projected/36fb9347-7d38-4da9-b135-643ebe867ef9-kube-api-access-kgjdb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:54.654865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.654828    4417 generic.go:356] "Generic (PLEG): container finished" podID="36fb9347-7d38-4da9-b135-643ebe867ef9" containerID="2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60" exitCode=0
Oct 31 16:25:54.655061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.654880    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"
Oct 31 16:25:54.655061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.654906    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" event={"ID":"36fb9347-7d38-4da9-b135-643ebe867ef9","Type":"ContainerDied","Data":"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"}
Oct 31 16:25:54.655061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.654961    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429" event={"ID":"36fb9347-7d38-4da9-b135-643ebe867ef9","Type":"ContainerDied","Data":"22f33b1c61f117038118d0969b9039940cc39499ba5d83540868b9d3aaa6ff37"}
Oct 31 16:25:54.655061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.655002    4417 scope.go:122] "RemoveContainer" containerID="2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"
Oct 31 16:25:54.665056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.665028    4417 scope.go:122] "RemoveContainer" containerID="2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"
Oct 31 16:25:54.665386 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:54.665363    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60\": container with ID starting with 2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60 not found: ID does not exist" containerID="2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"
Oct 31 16:25:54.665477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.665393    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60"} err="failed to get container status \"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60\": rpc error: code = NotFound desc = could not find container \"2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60\": container with ID starting with 2f960b0a9d0d1173db4932f80a422b8bca739855ae74a56693b36aeaa0ebcd60 not found: ID does not exist"
Oct 31 16:25:54.810766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.810716    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:25:54.856481 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:54.856447    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a08562e5-57698b7cdc-xj429"]
Oct 31 16:25:55.117638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:55.111458    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fb9347-7d38-4da9-b135-643ebe867ef9" path="/var/lib/kubelet/pods/36fb9347-7d38-4da9-b135-643ebe867ef9/volumes"
Oct 31 16:25:56.174664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.174561    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" podUID="47587759-3e3a-46a8-83da-d678b985404c" containerName="payload-processing" containerID="cri-o://71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd" gracePeriod=25
Oct 31 16:25:56.297027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.296964    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" podUID="768478da-1d69-4f3b-b90a-2e027a284d71" containerName="payload-pre-processing" containerID="cri-o://c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c" gracePeriod=25
Oct 31 16:25:56.481039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.481011    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:25:56.524290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.523264    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\" (UniqueName: \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\") pod \"47587759-3e3a-46a8-83da-d678b985404c\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") "
Oct 31 16:25:56.524290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.523371    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\") pod \"47587759-3e3a-46a8-83da-d678b985404c\" (UID: \"47587759-3e3a-46a8-83da-d678b985404c\") "
Oct 31 16:25:56.524531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.524298    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume" pod "47587759-3e3a-46a8-83da-d678b985404c" (UID: "47587759-3e3a-46a8-83da-d678b985404c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:56.527551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.527519    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd" pod "47587759-3e3a-46a8-83da-d678b985404c" (UID: "47587759-3e3a-46a8-83da-d678b985404c"). InnerVolumeSpecName "kube-api-access-ss9sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:56.579175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.579152    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:25:56.625579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.625542    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\") pod \"768478da-1d69-4f3b-b90a-2e027a284d71\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") "
Oct 31 16:25:56.625814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.625644    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\" (UniqueName: \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\") pod \"768478da-1d69-4f3b-b90a-2e027a284d71\" (UID: \"768478da-1d69-4f3b-b90a-2e027a284d71\") "
Oct 31 16:25:56.626684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.626655    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47587759-3e3a-46a8-83da-d678b985404c-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:56.626808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.626685    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9sd\" (UniqueName: \"kubernetes.io/projected/47587759-3e3a-46a8-83da-d678b985404c-kube-api-access-ss9sd\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:56.627309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.627227    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume" pod "768478da-1d69-4f3b-b90a-2e027a284d71" (UID: "768478da-1d69-4f3b-b90a-2e027a284d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:25:56.629976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.629942    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq" pod "768478da-1d69-4f3b-b90a-2e027a284d71" (UID: "768478da-1d69-4f3b-b90a-2e027a284d71"). InnerVolumeSpecName "kube-api-access-vszfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:25:56.674776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.674738    4417 generic.go:356] "Generic (PLEG): container finished" podID="768478da-1d69-4f3b-b90a-2e027a284d71" containerID="c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c" exitCode=0
Oct 31 16:25:56.674776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.674768    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" event={"ID":"768478da-1d69-4f3b-b90a-2e027a284d71","Type":"ContainerDied","Data":"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"}
Oct 31 16:25:56.675038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.674822    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh" event={"ID":"768478da-1d69-4f3b-b90a-2e027a284d71","Type":"ContainerDied","Data":"965693f2f445dfdc95264d213fbf92cca5f4f31838ffa9d40deead6889aceef1"}
Oct 31 16:25:56.675038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.674851    4417 scope.go:122] "RemoveContainer" containerID="c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"
Oct 31 16:25:56.675038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.674745    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"
Oct 31 16:25:56.676948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.676920    4417 generic.go:356] "Generic (PLEG): container finished" podID="47587759-3e3a-46a8-83da-d678b985404c" containerID="71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd" exitCode=0
Oct 31 16:25:56.677083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.676965    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"
Oct 31 16:25:56.677083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.677047    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" event={"ID":"47587759-3e3a-46a8-83da-d678b985404c","Type":"ContainerDied","Data":"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"}
Oct 31 16:25:56.677158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.677088    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x" event={"ID":"47587759-3e3a-46a8-83da-d678b985404c","Type":"ContainerDied","Data":"8696d50672aa5ffe8e071446e5ed2b2ff3f8c6d697f5c1b6ec850d3e34e0a96b"}
Oct 31 16:25:56.687132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.687109    4417 scope.go:122] "RemoveContainer" containerID="c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"
Oct 31 16:25:56.687605 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:56.687578    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c\": container with ID starting with c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c not found: ID does not exist" containerID="c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"
Oct 31 16:25:56.687730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.687645    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c"} err="failed to get container status \"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c\": rpc error: code = NotFound desc = could not find container \"c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c\": container with ID starting with c811088e524fc08e7abe6409b4826fb7d1145b54bb58b33439a561c29566944c not found: ID does not exist"
Oct 31 16:25:56.687730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.687668    4417 scope.go:122] "RemoveContainer" containerID="71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"
Oct 31 16:25:56.706668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.706644    4417 scope.go:122] "RemoveContainer" containerID="71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"
Oct 31 16:25:56.707032 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:25:56.707005    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd\": container with ID starting with 71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd not found: ID does not exist" containerID="71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"
Oct 31 16:25:56.707131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.707038    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd"} err="failed to get container status \"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd\": rpc error: code = NotFound desc = could not find container \"71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd\": container with ID starting with 71ba1b71538f493d0618d275b35003bb12e87ffae18f8ee2db36a5ef1e4045bd not found: ID does not exist"
Oct 31 16:25:56.727906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.727871    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vszfq\" (UniqueName: \"kubernetes.io/projected/768478da-1d69-4f3b-b90a-2e027a284d71-kube-api-access-vszfq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:56.727906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.727898    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/768478da-1d69-4f3b-b90a-2e027a284d71-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:25:56.796467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.796427    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:25:56.800608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.800559    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-caa664e1-857c657ccc-77m9x"]
Oct 31 16:25:56.901234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.901202    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:25:56.906150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:56.906094    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-caa664e1-979589fc5-jckbh"]
Oct 31 16:25:57.054140 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:57.054100    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47587759-3e3a-46a8-83da-d678b985404c" path="/var/lib/kubelet/pods/47587759-3e3a-46a8-83da-d678b985404c/volumes"
Oct 31 16:25:57.054741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:25:57.054718    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768478da-1d69-4f3b-b90a-2e027a284d71" path="/var/lib/kubelet/pods/768478da-1d69-4f3b-b90a-2e027a284d71/volumes"
Oct 31 16:26:02.950141 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:02.950100    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:26:15.902762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.902726    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:15.903707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903682    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b4109246-58f0-4de7-9b30-e269a2109ac9" containerName="payload-pre-processing"
Oct 31 16:26:15.903707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903706    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4109246-58f0-4de7-9b30-e269a2109ac9" containerName="payload-pre-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903719    4417 cpu_manager.go:414] "containerMap: removing container" podUID="36fb9347-7d38-4da9-b135-643ebe867ef9" containerName="payload-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903726    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fb9347-7d38-4da9-b135-643ebe867ef9" containerName="payload-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903745    4417 cpu_manager.go:414] "containerMap: removing container" podUID="768478da-1d69-4f3b-b90a-2e027a284d71" containerName="payload-pre-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903752    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768478da-1d69-4f3b-b90a-2e027a284d71" containerName="payload-pre-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903790    4417 cpu_manager.go:414] "containerMap: removing container" podUID="47587759-3e3a-46a8-83da-d678b985404c" containerName="payload-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903797    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47587759-3e3a-46a8-83da-d678b985404c" containerName="payload-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903834    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" containerName="payload-processing"
Oct 31 16:26:15.903901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.903841    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" containerName="payload-processing"
Oct 31 16:26:15.904276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.904031    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4109246-58f0-4de7-9b30-e269a2109ac9" containerName="payload-pre-processing"
Oct 31 16:26:15.904276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.904046    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fb9347-7d38-4da9-b135-643ebe867ef9" containerName="payload-processing"
Oct 31 16:26:15.904276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.904055    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="768478da-1d69-4f3b-b90a-2e027a284d71" containerName="payload-pre-processing"
Oct 31 16:26:15.904276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.904069    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="47587759-3e3a-46a8-83da-d678b985404c" containerName="payload-processing"
Oct 31 16:26:15.904276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.904084    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a803ee-edd8-4f38-a17e-69d93fdd32a0" containerName="payload-processing"
Oct 31 16:26:15.909170 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.909147    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.985692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.985650    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.985893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.985705    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.985952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.985889    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.985952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.985932    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmzsl\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.986142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.986100    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.986142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.986138    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.986343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.986169    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.986343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.986201    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:15.986343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:15.986251    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.087599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087558    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.087846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087648    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.087846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087688    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.087846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087717    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.087846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087769    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.087988    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088008    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088108    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088211    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088288    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088336    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmzsl\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088469    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088526    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.088728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.088683    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.090249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.090227    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.090658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.090636    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.107086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.107056    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:16.140282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.140249    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.140467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.140383    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmzsl\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.222950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.222863    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:16.650003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.649974    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:16.650402 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:16.650378    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d96df6_6d5c_456c_bc3d_7a90bb048bf1.slice/crio-391df2ee4d97070421e12ea364b3b4f86e0957787334d3481085cd21d4569f69 WatchSource:0}: Error finding container 391df2ee4d97070421e12ea364b3b4f86e0957787334d3481085cd21d4569f69: Status 404 returned error can't find the container with id 391df2ee4d97070421e12ea364b3b4f86e0957787334d3481085cd21d4569f69
Oct 31 16:26:16.653113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.653078    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:16.653189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.653145    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:16.653189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.653173    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:16.913088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.912994    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" event={"ID":"97d96df6-6d5c-456c-bc3d-7a90bb048bf1","Type":"ContainerStarted","Data":"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"}
Oct 31 16:26:16.913088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:16.913055    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" event={"ID":"97d96df6-6d5c-456c-bc3d-7a90bb048bf1","Type":"ContainerStarted","Data":"391df2ee4d97070421e12ea364b3b4f86e0957787334d3481085cd21d4569f69"}
Oct 31 16:26:17.022406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:17.022353    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podStartSLOduration=2.022336482 podStartE2EDuration="2.022336482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:17.020243244 +0000 UTC m=+1422.081818457" watchObservedRunningTime="2026-10-31 16:26:17.022336482 +0000 UTC m=+1422.083911668"
Oct 31 16:26:17.223077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:17.222986    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:17.224887 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:17.224864    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:17.224951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:17.224913    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:18.224058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:18.224020    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:18.224464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:18.224076    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:19.227355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:19.227318    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:19.228184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:19.228018    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:20.230532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:20.230479    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:20.231987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:20.231950    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:21.229646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:21.227066    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:21.229646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:21.227126    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:21.567643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:21.562848    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:26:21.567643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:21.563090    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy" containerID="cri-o://3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039" gracePeriod=30
Oct 31 16:26:22.225138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:22.225102    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:22.225444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:22.225404    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:22.683814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:22.658374    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:26:23.068635 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.068577    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:23.242798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.242748    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:23.415706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.375411    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:23.739881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.739745    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:26:23.745496 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.745329    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:23.761644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.747051    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:23.826668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.245135    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy" containerID="cri-o://da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d" gracePeriod=30
Oct 31 16:26:23.826668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.810217    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 31 16:26:23.827862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:23.827826    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 31 16:26:24.162638 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:24.161779    4417 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s"
Oct 31 16:26:24.246314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.244029    4417 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 31 16:26:24.246314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.244093    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 31 16:26:24.295302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.295267    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:26:24.299178 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.299133    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy" containerID="cri-o://5c2dbe4553bab41feaa6004cc52620aa723938baca3c65950e2896d105c586a0" gracePeriod=30
Oct 31 16:26:24.347641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.344325    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:24.411125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.411082    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:26:24.452356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.452250    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy" containerID="cri-o://9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca" gracePeriod=30
Oct 31 16:26:24.484027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.483972    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy" containerID="cri-o://f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf" gracePeriod=30
Oct 31 16:26:24.748722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.747795    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:24.748722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.748216    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 16:26:24.749293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.748850    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:26:24.749293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.748922    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:26:24.760705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.750852    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 16:26:24.760705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.751302    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:26:24.760705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.752123    4417 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:24.816866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.813921    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:24.816866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.814466    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:26:24.851772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.851723    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:24.852425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.852393    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:26:24.986649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:24.983773    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 31 16:26:25.207431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:25.207401    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:26:25.295962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:25.295928    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:26:25.306374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:25.306186    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:26:25.314200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:25.314162    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:26:25.318070 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:25.318020    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:26:26.112253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.112220    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:26:26.116806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.116784    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.178901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.178605    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ce4664d0-openshift-default-dockercfg-kwngg\""
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182255    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9vq\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182319    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182361    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182384    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182416    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182453    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.182647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182511    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.183414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182664    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.183414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.182729    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.206092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.206055    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340127    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340188    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340230    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340292    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340405    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340652    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340743    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340864    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9vq\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.340907    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.341846    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.342148    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.342503    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.343890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.343154    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.344491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.344425    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.373337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.360520    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.423210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.420495    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9vq\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.423210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.420788    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.424337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.424282    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\") pod \"e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") " pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.432999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.432962    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:26.851762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.851735    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:26:26.973439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973365    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973419    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973650    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973692    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973749    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973749    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:26.973911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973808    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:26.973911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973840    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.973911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973866    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.974089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.973935    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.974089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974017    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\") pod \"3713a7b3-2756-46ac-981e-dba2e031bf1d\" (UID: \"3713a7b3-2756-46ac-981e-dba2e031bf1d\") "
Oct 31 16:26:26.974200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974081    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:26.974200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974121    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:26.974441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974420    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:26.974814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974795    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:26.974902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974817    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:26.974902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974831    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3713a7b3-2756-46ac-981e-dba2e031bf1d-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:26.974902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974844    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:26.974902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.974858    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:26.976095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.976066    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:26.976193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.976104    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "kube-api-access-szs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:26.976280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.976259    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:26.976864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:26.976842    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token" pod "3713a7b3-2756-46ac-981e-dba2e031bf1d" (UID: "3713a7b3-2756-46ac-981e-dba2e031bf1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:27.006268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.006229    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:26:27.014972 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:27.014935    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb536711_6e8c_4881_98a2_6c379a48f3a1.slice/crio-278e8a47e3c52163fe47a704d5231705ef9802423f88f1c3195b0c346671dbe3 WatchSource:0}: Error finding container 278e8a47e3c52163fe47a704d5231705ef9802423f88f1c3195b0c346671dbe3: Status 404 returned error can't find the container with id 278e8a47e3c52163fe47a704d5231705ef9802423f88f1c3195b0c346671dbe3
Oct 31 16:26:27.018354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.018317    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:27.018452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.018403    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:27.018509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.018447    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:27.076989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.076951    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:27.076989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.076979    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:27.076989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.076992    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3713a7b3-2756-46ac-981e-dba2e031bf1d-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:27.077200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.077005    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szs62\" (UniqueName: \"kubernetes.io/projected/3713a7b3-2756-46ac-981e-dba2e031bf1d-kube-api-access-szs62\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:27.417445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.417406    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerStarted","Data":"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"}
Oct 31 16:26:27.417991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.417477    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerStarted","Data":"278e8a47e3c52163fe47a704d5231705ef9802423f88f1c3195b0c346671dbe3"}
Oct 31 16:26:27.419946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.419918    4417 generic.go:356] "Generic (PLEG): container finished" podID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerID="3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039" exitCode=0
Oct 31 16:26:27.420130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.420099    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerDied","Data":"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"}
Oct 31 16:26:27.420204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.420147    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb" event={"ID":"3713a7b3-2756-46ac-981e-dba2e031bf1d","Type":"ContainerDied","Data":"d6eaffd5302b20b19041e68df6861e47279711796b007e22a86915351a42fb92"}
Oct 31 16:26:27.420204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.420180    4417 scope.go:122] "RemoveContainer" containerID="3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"
Oct 31 16:26:27.420204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.420185    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"
Oct 31 16:26:27.433783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.433752    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:27.435129 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.435106    4417 scope.go:122] "RemoveContainer" containerID="89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"
Oct 31 16:26:27.445563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.445538    4417 scope.go:122] "RemoveContainer" containerID="3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"
Oct 31 16:26:27.445946 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:27.445926    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039\": container with ID starting with 3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039 not found: ID does not exist" containerID="3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"
Oct 31 16:26:27.446012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.445956    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039"} err="failed to get container status \"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039\": rpc error: code = NotFound desc = could not find container \"3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039\": container with ID starting with 3a44f3000bd987ef6dae5a5a81a11a6c5a750746314b1ad5f3830d5589d0b039 not found: ID does not exist"
Oct 31 16:26:27.446012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.445984    4417 scope.go:122] "RemoveContainer" containerID="89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"
Oct 31 16:26:27.446280 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:27.446261    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6\": container with ID starting with 89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6 not found: ID does not exist" containerID="89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"
Oct 31 16:26:27.446345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.446289    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6"} err="failed to get container status \"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6\": rpc error: code = NotFound desc = could not find container \"89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6\": container with ID starting with 89a4c06de06c0f53e3842823f53d9575738f9d18edb905c93ab201ee970cf8d6 not found: ID does not exist"
Oct 31 16:26:27.528673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.528597    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podStartSLOduration=1.528580925 podStartE2EDuration="1.528580925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:27.524283715 +0000 UTC m=+1432.585858901" watchObservedRunningTime="2026-10-31 16:26:27.528580925 +0000 UTC m=+1432.590156110"
Oct 31 16:26:27.629638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.629527    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:26:27.634114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:27.634077    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f27146d0-gw-openshift-default-5b4f7cdf6f-c8vdb"]
Oct 31 16:26:28.441098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:28.441064    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:29.084513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.084473    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" path="/var/lib/kubelet/pods/3713a7b3-2756-46ac-981e-dba2e031bf1d/volumes"
Oct 31 16:26:29.338420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.338356    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:26:29.432570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.432532    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.432752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.432580    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.432854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.432831    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433007    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433041    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433101    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.432837    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.433280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433156    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433230    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\") pod \"87f588d4-469c-4569-9cb5-7bb7163706f4\" (UID: \"87f588d4-469c-4569-9cb5-7bb7163706f4\") "
Oct 31 16:26:29.433437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433361    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.433494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433440    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.433748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.433717    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.434125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.434102    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.434218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.434131    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.434218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.434147    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.434218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.434161    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.434376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.434308    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:29.436060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.436031    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "kube-api-access-qz9dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.436163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.436085    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.436325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.436307    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:29.436388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.436345    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token" pod "87f588d4-469c-4569-9cb5-7bb7163706f4" (UID: "87f588d4-469c-4569-9cb5-7bb7163706f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.442033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.442005    4417 generic.go:356] "Generic (PLEG): container finished" podID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerID="da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d" exitCode=0
Oct 31 16:26:29.442431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.442076    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"
Oct 31 16:26:29.442431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.442066    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerDied","Data":"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"}
Oct 31 16:26:29.442431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.442177    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf" event={"ID":"87f588d4-469c-4569-9cb5-7bb7163706f4","Type":"ContainerDied","Data":"8de138f0ad31899a0328a5e3b472c65e6020e5463ec051a65f4a2260da15dc02"}
Oct 31 16:26:29.442431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.442213    4417 scope.go:122] "RemoveContainer" containerID="da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"
Oct 31 16:26:29.444604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.444586    4417 generic.go:356] "Generic (PLEG): container finished" podID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerID="5c2dbe4553bab41feaa6004cc52620aa723938baca3c65950e2896d105c586a0" exitCode=0
Oct 31 16:26:29.445786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.445739    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" event={"ID":"3beaed5d-9e83-4a3c-9f90-f038b05ad5de","Type":"ContainerDied","Data":"5c2dbe4553bab41feaa6004cc52620aa723938baca3c65950e2896d105c586a0"}
Oct 31 16:26:29.445883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.445815    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:29.447185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.447163    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:26:29.484147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.484118    4417 scope.go:122] "RemoveContainer" containerID="ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"
Oct 31 16:26:29.495248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.495228    4417 scope.go:122] "RemoveContainer" containerID="da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"
Oct 31 16:26:29.495595 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:29.495572    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d\": container with ID starting with da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d not found: ID does not exist" containerID="da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"
Oct 31 16:26:29.495720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.495604    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d"} err="failed to get container status \"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d\": rpc error: code = NotFound desc = could not find container \"da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d\": container with ID starting with da71da05a13f7547c708d067281d5281656659cc02d19668cd5e8aaa608cd70d not found: ID does not exist"
Oct 31 16:26:29.495720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.495633    4417 scope.go:122] "RemoveContainer" containerID="ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"
Oct 31 16:26:29.495923 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:29.495902    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2\": container with ID starting with ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2 not found: ID does not exist" containerID="ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"
Oct 31 16:26:29.495989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.495935    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2"} err="failed to get container status \"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2\": rpc error: code = NotFound desc = could not find container \"ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2\": container with ID starting with ee01016229c6aacbff9df9c93f956bfc57246d531db2067af748a30c28b0fac2 not found: ID does not exist"
Oct 31 16:26:29.535419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.535386    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.535419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.535419    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.535639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.535434    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.535639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.535446    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz9dh\" (UniqueName: \"kubernetes.io/projected/87f588d4-469c-4569-9cb5-7bb7163706f4-kube-api-access-qz9dh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.535639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.535458    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87f588d4-469c-4569-9cb5-7bb7163706f4-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.601500 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.601465    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:26:29.601992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.601965    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:26:29.606036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.605990    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a08562e5-openshift-default-c86877577-gv4tf"]
Oct 31 16:26:29.637303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.637225    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.637720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.637325    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.637720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.637428    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.637720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.637552    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.638250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.638226    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.638363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.637584    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.638459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.638442    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.638692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.638674    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.638781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.638766    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.639094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.639078    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.639179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.639119    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.639232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.639185    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\") pod \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\" (UID: \"3beaed5d-9e83-4a3c-9f90-f038b05ad5de\") "
Oct 31 16:26:29.639557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.639536    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.639841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.639821    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:29.640182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.640160    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.640272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.640185    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.640272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.640194    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.640272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.640203    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.640272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.640211    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.641324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.641297    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:29.642518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.642492    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "kube-api-access-g5cff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.651200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.651160    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.652036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.652006    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy" pod "3beaed5d-9e83-4a3c-9f90-f038b05ad5de" (UID: "3beaed5d-9e83-4a3c-9f90-f038b05ad5de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.715297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.715263    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:26:29.741425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741388    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.741702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741651    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.741702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741698    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.741859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741757    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.741859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741817    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.742026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741926    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.742026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.741967    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.742377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.742193    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.742377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.742233    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\") pod \"4528110c-2fa8-496f-b184-eabf72d7165c\" (UID: \"4528110c-2fa8-496f-b184-eabf72d7165c\") "
Oct 31 16:26:29.742841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.742584    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.742939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.742870    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743275    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743305    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5cff\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-kube-api-access-g5cff\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743320    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743335    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743349    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743362    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beaed5d-9e83-4a3c-9f90-f038b05ad5de-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.743664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.743475    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:29.744896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.744787    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.744896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.744870    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.746314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.746288    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:29.746429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.746406    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "kube-api-access-x6tgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.751308 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.751267    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:29.751428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.751318    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token" pod "4528110c-2fa8-496f-b184-eabf72d7165c" (UID: "4528110c-2fa8-496f-b184-eabf72d7165c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844609    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844686    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4528110c-2fa8-496f-b184-eabf72d7165c-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844697    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844710    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tgt\" (UniqueName: \"kubernetes.io/projected/4528110c-2fa8-496f-b184-eabf72d7165c-kube-api-access-x6tgt\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844723    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.844737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844731    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4528110c-2fa8-496f-b184-eabf72d7165c-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:29.845049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:29.844740    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4528110c-2fa8-496f-b184-eabf72d7165c-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:30.570639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.568684    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.648851    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672737    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672763    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672796    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672805    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672829    4417 cpu_manager.go:414] "containerMap: removing container" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672836    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672864    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672871    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672887    4417 cpu_manager.go:414] "containerMap: removing container" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672894    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672916    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.672924    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.673187    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.673208    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.673220    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.673230    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.666277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.673244    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" containerName="istio-proxy"
Oct 31 16:26:30.740642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.732348    4417 generic.go:356] "Generic (PLEG): container finished" podID="4528110c-2fa8-496f-b184-eabf72d7165c" containerID="9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca" exitCode=0
Oct 31 16:26:30.740642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.733403    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"
Oct 31 16:26:30.850639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.848058    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6" event={"ID":"3beaed5d-9e83-4a3c-9f90-f038b05ad5de","Type":"ContainerDied","Data":"15f57ba6149c2bd11d0c1a1252d48c8873a4ca8815b6c2c3db1caa033490b33d"}
Oct 31 16:26:30.850639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.848121    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerDied","Data":"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"}
Oct 31 16:26:30.850639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.848142    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4" event={"ID":"4528110c-2fa8-496f-b184-eabf72d7165c","Type":"ContainerDied","Data":"dc8cf138a443807a089ffd0b5344762779c37b0e80de93eb91cb07f3624e92a5"}
Oct 31 16:26:30.850639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.848164    4417 scope.go:122] "RemoveContainer" containerID="5c2dbe4553bab41feaa6004cc52620aa723938baca3c65950e2896d105c586a0"
Oct 31 16:26:30.850639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.848360    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:30.883256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.881883    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:26:30.897702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.897676    4417 scope.go:122] "RemoveContainer" containerID="9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"
Oct 31 16:26:30.898210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.898190    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c1bb9544\""
Oct 31 16:26:30.898459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.898445    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c1bb9544-dockercfg-d6f7g\""
Oct 31 16:26:30.898734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.898720    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:26:30.905602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.905578    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.905768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.905754    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3713a7b3-2756-46ac-981e-dba2e031bf1d" containerName="istio-proxy"
Oct 31 16:26:30.906128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.906115    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" containerName="istio-proxy"
Oct 31 16:26:30.906212 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.906203    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" containerName="istio-proxy"
Oct 31 16:26:30.911027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.910976    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:26:30.911115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.911083    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:30.925463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.925439    4417 scope.go:122] "RemoveContainer" containerID="afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"
Oct 31 16:26:30.940936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.940911    4417 scope.go:122] "RemoveContainer" containerID="9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"
Oct 31 16:26:30.941290 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:30.941269    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca\": container with ID starting with 9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca not found: ID does not exist" containerID="9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"
Oct 31 16:26:30.941366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.941306    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca"} err="failed to get container status \"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca\": rpc error: code = NotFound desc = could not find container \"9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca\": container with ID starting with 9e4f24f72f3284de6955b5f587a354b6fca96fe8ec293473ed47269c94df47ca not found: ID does not exist"
Oct 31 16:26:30.941366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.941335    4417 scope.go:122] "RemoveContainer" containerID="afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"
Oct 31 16:26:30.941636 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:30.941605    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8\": container with ID starting with afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8 not found: ID does not exist" containerID="afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"
Oct 31 16:26:30.941700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.941648    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8"} err="failed to get container status \"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8\": rpc error: code = NotFound desc = could not find container \"afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8\": container with ID starting with afc3a6d59411785202352378f3423a32d866ed5198220811247ba8ef226036f8 not found: ID does not exist"
Oct 31 16:26:30.949986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.949956    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:26:30.954371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:30.954345    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-vcnp4"]
Oct 31 16:26:31.008173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.008125    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.008327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.008206    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvvt\" (UniqueName: \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.056345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.056311    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4528110c-2fa8-496f-b184-eabf72d7165c" path="/var/lib/kubelet/pods/4528110c-2fa8-496f-b184-eabf72d7165c/volumes"
Oct 31 16:26:31.059674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.057142    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f588d4-469c-4569-9cb5-7bb7163706f4" path="/var/lib/kubelet/pods/87f588d4-469c-4569-9cb5-7bb7163706f4/volumes"
Oct 31 16:26:31.059869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.059844    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:26:31.062929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.062907    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-gctb6"]
Oct 31 16:26:31.109843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.109809    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgd2n\" (UniqueName: \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.110045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.109935    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.110045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.109993    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvvt\" (UniqueName: \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.110247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.110225    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.110772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.110748    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.169948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.169916    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvvt\" (UniqueName: \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\") pod \"payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.198974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.198943    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:31.200747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.200727    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.212761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.212724    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.212895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.212878    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgd2n\" (UniqueName: \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.214640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.214548    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.277666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.272525    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgd2n\" (UniqueName: \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\") pod \"payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") " pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.314356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314320    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314385    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314414    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314448    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314545    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314709    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314837    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.314955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.314930    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.315049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.315031    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\") pod \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\" (UID: \"97d96df6-6d5c-456c-bc3d-7a90bb048bf1\") "
Oct 31 16:26:31.316375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.316033    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:31.316375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.316057    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:31.316663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.316499    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:31.316731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.316662    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:31.317060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.317038    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:31.318389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.318355    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:31.318828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.318791    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:31.319307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.319284    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:31.319424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.319400    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl" pod "97d96df6-6d5c-456c-bc3d-7a90bb048bf1" (UID: "97d96df6-6d5c-456c-bc3d-7a90bb048bf1"). InnerVolumeSpecName "kube-api-access-vmzsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417752    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417780    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417789    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417798    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417809    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417817    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmzsl\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-kube-api-access-vmzsl\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417827    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.417832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417836    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.418273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.417844    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d96df6-6d5c-456c-bc3d-7a90bb048bf1-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:31.528173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.528131    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:31.586363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.585899    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:26:31.743068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.743032    4417 generic.go:356] "Generic (PLEG): container finished" podID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerID="f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf" exitCode=0
Oct 31 16:26:31.743309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.743288    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"
Oct 31 16:26:31.743397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.743248    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" event={"ID":"97d96df6-6d5c-456c-bc3d-7a90bb048bf1","Type":"ContainerDied","Data":"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"}
Oct 31 16:26:31.743460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.743434    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf" event={"ID":"97d96df6-6d5c-456c-bc3d-7a90bb048bf1","Type":"ContainerDied","Data":"391df2ee4d97070421e12ea364b3b4f86e0957787334d3481085cd21d4569f69"}
Oct 31 16:26:31.743522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.743475    4417 scope.go:122] "RemoveContainer" containerID="f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"
Oct 31 16:26:31.748453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.748422    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" event={"ID":"1483bf0c-7ece-4838-9c45-3c936a6c0cd9","Type":"ContainerStarted","Data":"a0291b89a57b030ae0682b41fb0afe1b8d0a9d83ce977789d172dbd5a49d7efd"}
Oct 31 16:26:31.748612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.748480    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" event={"ID":"1483bf0c-7ece-4838-9c45-3c936a6c0cd9","Type":"ContainerStarted","Data":"26377898ef66f3c4a99f2e7c6863149a96d9203211ccc462b9f081c46b061236"}
Oct 31 16:26:31.748612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.748513    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:31.757338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.757320    4417 scope.go:122] "RemoveContainer" containerID="f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"
Oct 31 16:26:31.757612 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:31.757593    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf\": container with ID starting with f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf not found: ID does not exist" containerID="f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"
Oct 31 16:26:31.757705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.757650    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf"} err="failed to get container status \"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf\": rpc error: code = NotFound desc = could not find container \"f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf\": container with ID starting with f8114dca286e49a247e5c855c4820353afb2701c3f1fefc47f7d917e661601bf not found: ID does not exist"
Oct 31 16:26:31.898232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.898173    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" podStartSLOduration=1.898157055 podStartE2EDuration="1.898157055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:31.896897018 +0000 UTC m=+1436.958472205" watchObservedRunningTime="2026-10-31 16:26:31.898157055 +0000 UTC m=+1436.959732241"
Oct 31 16:26:31.905996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:31.905966    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:26:31.943604 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:31.943574    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d480b1_e162_4ad8_8aca_1c51def2da96.slice/crio-c7a6cccfb670498b32c473f4225c3a44c5eaaa6670c512c331e548aaf4cd2dc6 WatchSource:0}: Error finding container c7a6cccfb670498b32c473f4225c3a44c5eaaa6670c512c331e548aaf4cd2dc6: Status 404 returned error can't find the container with id c7a6cccfb670498b32c473f4225c3a44c5eaaa6670c512c331e548aaf4cd2dc6
Oct 31 16:26:32.006732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.006688    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:32.008690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.008661    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-ngbsf"]
Oct 31 16:26:32.763699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.763658    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" event={"ID":"e4d480b1-e162-4ad8-8aca-1c51def2da96","Type":"ContainerStarted","Data":"54896effd3a0eaa5e6a9ac0a3cba89d27edd8565bdce8f7e5e5b2749ae9dcf64"}
Oct 31 16:26:32.764188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.763724    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" event={"ID":"e4d480b1-e162-4ad8-8aca-1c51def2da96","Type":"ContainerStarted","Data":"c7a6cccfb670498b32c473f4225c3a44c5eaaa6670c512c331e548aaf4cd2dc6"}
Oct 31 16:26:32.764188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.763923    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:32.870478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:32.870419    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" podStartSLOduration=2.870403052 podStartE2EDuration="2.870403052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:32.867862289 +0000 UTC m=+1437.929437474" watchObservedRunningTime="2026-10-31 16:26:32.870403052 +0000 UTC m=+1437.931978238"
Oct 31 16:26:32.945698 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:32.945657    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:26:33.075047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:33.074995    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3beaed5d-9e83-4a3c-9f90-f038b05ad5de" path="/var/lib/kubelet/pods/3beaed5d-9e83-4a3c-9f90-f038b05ad5de/volumes"
Oct 31 16:26:33.078651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:33.075926    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" path="/var/lib/kubelet/pods/97d96df6-6d5c-456c-bc3d-7a90bb048bf1/volumes"
Oct 31 16:26:34.878735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.877672    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:26:34.880661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.879382    4417 cpu_manager.go:414] "containerMap: removing container" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy"
Oct 31 16:26:34.880661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.879407    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy"
Oct 31 16:26:34.880661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.879648    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d96df6-6d5c-456c-bc3d-7a90bb048bf1" containerName="istio-proxy"
Oct 31 16:26:34.905701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.897934    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:34.977642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:34.973472    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:26:35.029232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.029192    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:26:35.044875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.039162    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ce4664d0-dockercfg-b7n2h\""
Oct 31 16:26:35.162179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.158349    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ce4664d0\""
Oct 31 16:26:35.224116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.224082    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.224699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.224679    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz2f\" (UniqueName: \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.304705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.304671    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:26:35.406279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.309154    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.406279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.330678    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.406279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.384721    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz2f\" (UniqueName: \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.406279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.350260    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.454664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.451595    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz2f\" (UniqueName: \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\") pod \"payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.486141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.486108    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.486277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.486256    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.587418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.587378    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.587576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.587469    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.588137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.588118    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.640170 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.640130    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\") pod \"payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") " pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:35.689488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.689453    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:35.780826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:35.780493    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:36.072656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.072610    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:26:36.073092 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:36.072777    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee53f8ce_12e6_45c6_bb5e_e8bdf0aa54f4.slice/crio-2f63d9d2b119f095d1075abb4750905bcddbd891e4db593f7170ccc7eb6d5a71 WatchSource:0}: Error finding container 2f63d9d2b119f095d1075abb4750905bcddbd891e4db593f7170ccc7eb6d5a71: Status 404 returned error can't find the container with id 2f63d9d2b119f095d1075abb4750905bcddbd891e4db593f7170ccc7eb6d5a71
Oct 31 16:26:36.174794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.174761    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:26:36.176871 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:36.176847    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ebaca5_81af_48b7_b77a_1e313fb10505.slice/crio-f7fb4de65feb3a290d7f34578a056cdabcdabc9effb73d496a891778d0a636cb WatchSource:0}: Error finding container f7fb4de65feb3a290d7f34578a056cdabcdabc9effb73d496a891778d0a636cb: Status 404 returned error can't find the container with id f7fb4de65feb3a290d7f34578a056cdabcdabc9effb73d496a891778d0a636cb
Oct 31 16:26:36.914919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.914867    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" event={"ID":"06ebaca5-81af-48b7-b77a-1e313fb10505","Type":"ContainerStarted","Data":"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"}
Oct 31 16:26:36.915112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.914942    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" event={"ID":"06ebaca5-81af-48b7-b77a-1e313fb10505","Type":"ContainerStarted","Data":"f7fb4de65feb3a290d7f34578a056cdabcdabc9effb73d496a891778d0a636cb"}
Oct 31 16:26:36.915269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.915245    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:36.916767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.916741    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" event={"ID":"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4","Type":"ContainerStarted","Data":"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"}
Oct 31 16:26:36.916880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.916787    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" event={"ID":"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4","Type":"ContainerStarted","Data":"2f63d9d2b119f095d1075abb4750905bcddbd891e4db593f7170ccc7eb6d5a71"}
Oct 31 16:26:36.916880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:36.916824    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:37.022216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:37.022164    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" podStartSLOduration=3.022149587 podStartE2EDuration="3.022149587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:37.019130444 +0000 UTC m=+1442.080705631" watchObservedRunningTime="2026-10-31 16:26:37.022149587 +0000 UTC m=+1442.083724772"
Oct 31 16:26:37.125314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:37.125266    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" podStartSLOduration=3.125252239 podStartE2EDuration="3.125252239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:37.122267504 +0000 UTC m=+1442.183842700" watchObservedRunningTime="2026-10-31 16:26:37.125252239 +0000 UTC m=+1442.186827458"
Oct 31 16:26:42.767329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:42.767298    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:26:43.774253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:43.774222    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:26:47.581763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:47.581713    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:26:47.582545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:47.582504    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:26:47.930805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:47.930706    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:26:47.931429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:47.931404    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:26:52.988237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:52.988197    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:26:52.988782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:52.988427    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy" containerID="cri-o://fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084" gracePeriod=30
Oct 31 16:26:53.813430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.813385    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:26:53.818240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.818215    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.847962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.847926    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57mb\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.847972    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.848118    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.848329    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.848419    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.848527    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.848674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.848657    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.928177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.928134    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:26:53.950023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.949991    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950060    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950141    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950234    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950294    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57mb\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950334    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.950493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.950421    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.951473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.951405    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.953557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.953529    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.953731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.953693    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.957206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.957183    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:53.959549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:53.959506    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:54.003485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.003449    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:54.004820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.004791    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57mb\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzm9d\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:54.132851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.132762    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:54.558069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.558033    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:26:54.559996 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:26:54.559964    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d71e33_c1dc_4fea_ad9d_344f28388051.slice/crio-d101fc2a60ad4ad9601411f449822e48558eff0f2b26ef76d0df93f7da827f60 WatchSource:0}: Error finding container d101fc2a60ad4ad9601411f449822e48558eff0f2b26ef76d0df93f7da827f60: Status 404 returned error can't find the container with id d101fc2a60ad4ad9601411f449822e48558eff0f2b26ef76d0df93f7da827f60
Oct 31 16:26:54.562776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.562743    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:54.562889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:54.562806    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:26:55.336047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.335946    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" event={"ID":"86d71e33-c1dc-4fea-ad9d-344f28388051","Type":"ContainerStarted","Data":"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"}
Oct 31 16:26:55.336047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.336037    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" event={"ID":"86d71e33-c1dc-4fea-ad9d-344f28388051","Type":"ContainerStarted","Data":"d101fc2a60ad4ad9601411f449822e48558eff0f2b26ef76d0df93f7da827f60"}
Oct 31 16:26:55.336673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.336084    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:26:55.338002 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.337974    4417 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lzm9d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:26:55.338117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.338029    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:26:55.527798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:55.527745    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" podStartSLOduration=2.527728828 podStartE2EDuration="2.527728828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:55.488675158 +0000 UTC m=+1460.550250341" watchObservedRunningTime="2026-10-31 16:26:55.527728828 +0000 UTC m=+1460.589304013"
Oct 31 16:26:56.344120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:56.344090    4417 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lzm9d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:26:56.344535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:56.344143    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:26:58.281169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.281143    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:26:58.358642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.358578    4417 generic.go:356] "Generic (PLEG): container finished" podID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerID="fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084" exitCode=0
Oct 31 16:26:58.358820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.358652    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"
Oct 31 16:26:58.358820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.358670    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" event={"ID":"1804c48b-5202-48a6-be6e-06b0307f2ba8","Type":"ContainerDied","Data":"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"}
Oct 31 16:26:58.358820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.358722    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd" event={"ID":"1804c48b-5202-48a6-be6e-06b0307f2ba8","Type":"ContainerDied","Data":"7bb603ab0297ac53296774170888e5c6ac19df88f74ee2043e11dfc1aea275c5"}
Oct 31 16:26:58.358820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.358745    4417 scope.go:122] "RemoveContainer" containerID="fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"
Oct 31 16:26:58.370214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.370194    4417 scope.go:122] "RemoveContainer" containerID="fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"
Oct 31 16:26:58.412718 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:26:58.370480    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084\": container with ID starting with fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084 not found: ID does not exist" containerID="fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"
Oct 31 16:26:58.412718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.370513    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084"} err="failed to get container status \"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084\": rpc error: code = NotFound desc = could not find container \"fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084\": container with ID starting with fdecfcee422d78f66fabf8c019e0e6680c4c4c334d7bf2c674c66543f926f084 not found: ID does not exist"
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437223    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437375    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437490    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437557    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437635    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437715    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437771    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437880    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.437908    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\") pod \"1804c48b-5202-48a6-be6e-06b0307f2ba8\" (UID: \"1804c48b-5202-48a6-be6e-06b0307f2ba8\") "
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.440016    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.440305    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.441847    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.441876    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.442336    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.442643    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.445937    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "kube-api-access-7rrc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.446163    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:58.504905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.489934    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:26:58.537959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.537867    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:26:58.540524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.540497    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy" pod "1804c48b-5202-48a6-be6e-06b0307f2ba8" (UID: "1804c48b-5202-48a6-be6e-06b0307f2ba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544117    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544149    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544165    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rrc7\" (UniqueName: \"kubernetes.io/projected/1804c48b-5202-48a6-be6e-06b0307f2ba8-kube-api-access-7rrc7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544181    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1804c48b-5202-48a6-be6e-06b0307f2ba8-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544197    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544210    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.545775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.544223    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1804c48b-5202-48a6-be6e-06b0307f2ba8-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:26:58.773708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.773666    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:26:58.781228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:58.781193    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-d6kwd"]
Oct 31 16:26:59.051214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:59.051180    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" path="/var/lib/kubelet/pods/1804c48b-5202-48a6-be6e-06b0307f2ba8/volumes"
Oct 31 16:26:59.344588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:26:59.344555    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:27:02.947695 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:02.947659    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:27:07.627064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.627030    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:27:07.627812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.627789    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy"
Oct 31 16:27:07.627812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.627809    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy"
Oct 31 16:27:07.627986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.627932    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1804c48b-5202-48a6-be6e-06b0307f2ba8" containerName="istio-proxy"
Oct 31 16:27:07.633250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.633231    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.681387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.681352    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c1bb9544-serving-cert\""
Oct 31 16:27:07.720029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.719988    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.720226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.720148    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.720367 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.720344    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkb9\" (UniqueName: \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.720468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.720454    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.731332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.731301    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:27:07.822140 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.822104    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.822352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.822216    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.822352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.822315    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkb9\" (UniqueName: \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.822484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.822414    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.822582 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:07.822562    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-serving-cert: secret "maas-api-e2e-ait-c1bb9544-serving-cert" not found
Oct 31 16:27:07.822735 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:07.822657    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls podName:e0176593-9391-4023-81cc-0c406b5ec2d7 nodeName:}" failed. No retries permitted until 2026-10-31 16:27:08.322605795 +0000 UTC m=+1473.384180979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls") pod "maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" (UID: "e0176593-9391-4023-81cc-0c406b5ec2d7") : secret "maas-api-e2e-ait-c1bb9544-serving-cert" not found
Oct 31 16:27:07.823419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.823394    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.826036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.826012    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:07.878922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:07.878842    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkb9\" (UniqueName: \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:08.385678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:08.378139    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:08.519876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:08.519830    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") pod \"maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:08.605716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:08.576981    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:09.169849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:09.169815    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:27:09.171688 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:27:09.171660    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0176593_9391_4023_81cc_0c406b5ec2d7.slice/crio-59b63b69deb116c23da655a3770f69652e1e4b9deb9faa379d816c34ddce9b8f WatchSource:0}: Error finding container 59b63b69deb116c23da655a3770f69652e1e4b9deb9faa379d816c34ddce9b8f: Status 404 returned error can't find the container with id 59b63b69deb116c23da655a3770f69652e1e4b9deb9faa379d816c34ddce9b8f
Oct 31 16:27:09.596393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:09.596348    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" event={"ID":"e0176593-9391-4023-81cc-0c406b5ec2d7","Type":"ContainerStarted","Data":"59b63b69deb116c23da655a3770f69652e1e4b9deb9faa379d816c34ddce9b8f"}
Oct 31 16:27:10.606853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:10.606808    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" event={"ID":"e0176593-9391-4023-81cc-0c406b5ec2d7","Type":"ContainerStarted","Data":"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"}
Oct 31 16:27:10.607270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:10.606902    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:10.716545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:10.716489    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" podStartSLOduration=2.768589904 podStartE2EDuration="3.716472988s" totalImagesPullingTime="947.883084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:10.713521041 +0000 UTC m=+1475.775096230" watchObservedRunningTime="2026-10-31 16:27:10.716472988 +0000 UTC m=+1475.778048173"
Oct 31 16:27:12.703583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.703543    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:27:12.708373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.708344    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.759467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.759432    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ce4664d0-serving-cert\""
Oct 31 16:27:12.809173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.809132    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:27:12.853156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.853115    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.853346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.853177    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.853395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.853380    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.853449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.853437    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtq4m\" (UniqueName: \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.955558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.955449    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.955776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.955563    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtq4m\" (UniqueName: \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.955776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.955680    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.955776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.955734    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.955977 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:12.955928    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-serving-cert: secret "maas-api-e2e-ait-ce4664d0-serving-cert" not found
Oct 31 16:27:12.956036 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:12.956011    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls podName:e06c87d2-39aa-484b-b7cc-708e915319f6 nodeName:}" failed. No retries permitted until 2026-10-31 16:27:13.455991129 +0000 UTC m=+1478.517566307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls") pod "maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" (UID: "e06c87d2-39aa-484b-b7cc-708e915319f6") : secret "maas-api-e2e-ait-ce4664d0-serving-cert" not found
Oct 31 16:27:12.956475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.956444    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:12.958193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:12.958173    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:13.016667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:13.010345    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtq4m\" (UniqueName: \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:13.515658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:13.515574    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:13.518132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:13.518107    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") pod \"maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:13.622533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:13.622491    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:13.997591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:13.997560    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:27:13.998961 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:27:13.998931    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06c87d2_39aa_484b_b7cc_708e915319f6.slice/crio-e7c046c8cbb2ff5062efd497297c6c9cd6360a0efeb62064afbd5add5b52805b WatchSource:0}: Error finding container e7c046c8cbb2ff5062efd497297c6c9cd6360a0efeb62064afbd5add5b52805b: Status 404 returned error can't find the container with id e7c046c8cbb2ff5062efd497297c6c9cd6360a0efeb62064afbd5add5b52805b
Oct 31 16:27:14.636352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:14.636307    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" event={"ID":"e06c87d2-39aa-484b-b7cc-708e915319f6","Type":"ContainerStarted","Data":"e7c046c8cbb2ff5062efd497297c6c9cd6360a0efeb62064afbd5add5b52805b"}
Oct 31 16:27:15.646935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:15.646894    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" event={"ID":"e06c87d2-39aa-484b-b7cc-708e915319f6","Type":"ContainerStarted","Data":"9f352d76c4b58a9d181b9f6b84ee056d1f75be2b9997ca5b8faac528968c9b90"}
Oct 31 16:27:15.647330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:15.646987    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:15.756561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:15.756495    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" podStartSLOduration=2.806744727 podStartE2EDuration="3.756476982s" totalImagesPullingTime="949.732255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:15.752927876 +0000 UTC m=+1480.814503063" watchObservedRunningTime="2026-10-31 16:27:15.756476982 +0000 UTC m=+1480.818052169"
Oct 31 16:27:16.619504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:16.619471    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:27:21.658829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:21.658797    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:27:26.928708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:26.928666    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:27:27.033646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.004277    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.148474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.148439    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a7907d70-dockercfg-vr5gc\""
Oct 31 16:27:27.164641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.148769    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a7907d70\""
Oct 31 16:27:27.184164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.184072    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvzj\" (UniqueName: \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.185011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.184984    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.291653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.290308    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvzj\" (UniqueName: \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.291653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.290587    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.326939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.307807    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.363105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.356476    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvzj\" (UniqueName: \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\") pod \"payload-processing-e2e-mt-a7907d70-f46b98989-sb697\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") " pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.363105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.360005    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:27:27.469145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.468995    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:27.872747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.872711    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:27:27.874249 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:27:27.874203    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b385f77_5319_469c_acd9_a4336c384293.slice/crio-225db0ba3f88b49ef8e400b087c1aceb815cb1ab8e13f0249baf8b5eb113a699 WatchSource:0}: Error finding container 225db0ba3f88b49ef8e400b087c1aceb815cb1ab8e13f0249baf8b5eb113a699: Status 404 returned error can't find the container with id 225db0ba3f88b49ef8e400b087c1aceb815cb1ab8e13f0249baf8b5eb113a699
Oct 31 16:27:27.993169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.993129    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" event={"ID":"6b385f77-5319-469c-acd9-a4336c384293","Type":"ContainerStarted","Data":"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"}
Oct 31 16:27:27.993708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.993189    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" event={"ID":"6b385f77-5319-469c-acd9-a4336c384293","Type":"ContainerStarted","Data":"225db0ba3f88b49ef8e400b087c1aceb815cb1ab8e13f0249baf8b5eb113a699"}
Oct 31 16:27:27.993708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:27.993217    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:27:28.097983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:28.097934    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" podStartSLOduration=2.097918477 podStartE2EDuration="2.097918477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:28.09689113 +0000 UTC m=+1493.158466317" watchObservedRunningTime="2026-10-31 16:27:28.097918477 +0000 UTC m=+1493.159493663"
Oct 31 16:27:32.945747 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:27:32.945710    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:27:39.003468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:27:39.003430    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:28:00.486852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:00.486806    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:28:00.487414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:00.486899    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:01.926071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.926038    4417 generic.go:356] "Generic (PLEG): container finished" podID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerID="b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb" exitCode=0
Oct 31 16:28:01.926493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.926091    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerDied","Data":"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"}
Oct 31 16:28:01.926580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.926563    4417 scope.go:122] "RemoveContainer" containerID="b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"
Oct 31 16:28:01.928297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.928265    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:01.928373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.928340    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:01.928421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:01.928383    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:02.942083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:02.941010    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerStarted","Data":"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"}
Oct 31 16:28:02.953944 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:28:02.953912    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:28:03.438389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:03.438355    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:28:03.438781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:03.438756    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:03.438963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:03.438939    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:04.450642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:04.450042    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:04.450642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:04.450113    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:05.434045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:05.434006    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:05.434385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:05.434341    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:06.459011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:06.458976    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:06.459451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:06.459030    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:07.443071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:07.443034    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:07.443380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:07.443351    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:08.437015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:08.436957    4417 patch_prober.go:29] interesting pod/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 16:28:08.437844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:08.437811    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 16:28:09.438532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:09.438500    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:28:09.438956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:09.438886    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:28:09.439552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:09.439531    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:28:21.909000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:21.908966    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:28:22.010246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:22.010212    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:28:26.913777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:26.913711    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" podUID="e4d480b1-e162-4ad8-8aca-1c51def2da96" containerName="payload-processing" containerID="cri-o://54896effd3a0eaa5e6a9ac0a3cba89d27edd8565bdce8f7e5e5b2749ae9dcf64" gracePeriod=25
Oct 31 16:28:27.010725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.010673    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" podUID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" containerName="payload-pre-processing" containerID="cri-o://a0291b89a57b030ae0682b41fb0afe1b8d0a9d83ce977789d172dbd5a49d7efd" gracePeriod=25
Oct 31 16:28:27.458705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.449855    4417 generic.go:356] "Generic (PLEG): container finished" podID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" containerID="a0291b89a57b030ae0682b41fb0afe1b8d0a9d83ce977789d172dbd5a49d7efd" exitCode=0
Oct 31 16:28:27.458705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.449971    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" event={"ID":"1483bf0c-7ece-4838-9c45-3c936a6c0cd9","Type":"ContainerDied","Data":"a0291b89a57b030ae0682b41fb0afe1b8d0a9d83ce977789d172dbd5a49d7efd"}
Oct 31 16:28:27.458705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.452397    4417 generic.go:356] "Generic (PLEG): container finished" podID="e4d480b1-e162-4ad8-8aca-1c51def2da96" containerID="54896effd3a0eaa5e6a9ac0a3cba89d27edd8565bdce8f7e5e5b2749ae9dcf64" exitCode=0
Oct 31 16:28:27.458705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.452884    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" event={"ID":"e4d480b1-e162-4ad8-8aca-1c51def2da96","Type":"ContainerDied","Data":"54896effd3a0eaa5e6a9ac0a3cba89d27edd8565bdce8f7e5e5b2749ae9dcf64"}
Oct 31 16:28:27.933703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:27.931755    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:28:28.201085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.201007    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:28:28.299853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.299816    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\" (UniqueName: \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\") pod \"e4d480b1-e162-4ad8-8aca-1c51def2da96\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") "
Oct 31 16:28:28.330512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.330484    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\") pod \"e4d480b1-e162-4ad8-8aca-1c51def2da96\" (UID: \"e4d480b1-e162-4ad8-8aca-1c51def2da96\") "
Oct 31 16:28:28.332169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.316815    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n" pod "e4d480b1-e162-4ad8-8aca-1c51def2da96" (UID: "e4d480b1-e162-4ad8-8aca-1c51def2da96"). InnerVolumeSpecName "kube-api-access-sgd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:28.332169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.331050    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume" pod "e4d480b1-e162-4ad8-8aca-1c51def2da96" (UID: "e4d480b1-e162-4ad8-8aca-1c51def2da96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:28.332580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.332558    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgd2n\" (UniqueName: \"kubernetes.io/projected/e4d480b1-e162-4ad8-8aca-1c51def2da96-kube-api-access-sgd2n\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:28.332819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.332802    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d480b1-e162-4ad8-8aca-1c51def2da96-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:28.464643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.464528    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:28:28.467603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.467570    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48" event={"ID":"1483bf0c-7ece-4838-9c45-3c936a6c0cd9","Type":"ContainerDied","Data":"26377898ef66f3c4a99f2e7c6863149a96d9203211ccc462b9f081c46b061236"}
Oct 31 16:28:28.467877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.467858    4417 scope.go:122] "RemoveContainer" containerID="a0291b89a57b030ae0682b41fb0afe1b8d0a9d83ce977789d172dbd5a49d7efd"
Oct 31 16:28:28.473296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.473272    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj" event={"ID":"e4d480b1-e162-4ad8-8aca-1c51def2da96","Type":"ContainerDied","Data":"c7a6cccfb670498b32c473f4225c3a44c5eaaa6670c512c331e548aaf4cd2dc6"}
Oct 31 16:28:28.473959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.473939    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"
Oct 31 16:28:28.533528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.533464    4417 scope.go:122] "RemoveContainer" containerID="54896effd3a0eaa5e6a9ac0a3cba89d27edd8565bdce8f7e5e5b2749ae9dcf64"
Oct 31 16:28:28.608682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.557131    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\") pod \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") "
Oct 31 16:28:28.608682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.557473    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\" (UniqueName: \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\") pod \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\" (UID: \"1483bf0c-7ece-4838-9c45-3c936a6c0cd9\") "
Oct 31 16:28:28.608682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.566857    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume" pod "1483bf0c-7ece-4838-9c45-3c936a6c0cd9" (UID: "1483bf0c-7ece-4838-9c45-3c936a6c0cd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:28.614918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.612679    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt" pod "1483bf0c-7ece-4838-9c45-3c936a6c0cd9" (UID: "1483bf0c-7ece-4838-9c45-3c936a6c0cd9"). InnerVolumeSpecName "kube-api-access-htvvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:28.668075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.668030    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:28.668646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.668599    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htvvt\" (UniqueName: \"kubernetes.io/projected/1483bf0c-7ece-4838-9c45-3c936a6c0cd9-kube-api-access-htvvt\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:28.727059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.726964    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:28:28.731113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:28.731086    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1bb9544-7567c84675-xrsbj"]
Oct 31 16:28:29.124650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:29.124262    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d480b1-e162-4ad8-8aca-1c51def2da96" path="/var/lib/kubelet/pods/e4d480b1-e162-4ad8-8aca-1c51def2da96/volumes"
Oct 31 16:28:29.488319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:29.488236    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"
Oct 31 16:28:29.644866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:29.644832    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:28:29.688571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:29.688534    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1bb9544-55c58b8c4c-mtq48"]
Oct 31 16:28:31.051418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:31.051385    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" path="/var/lib/kubelet/pods/1483bf0c-7ece-4838-9c45-3c936a6c0cd9/volumes"
Oct 31 16:28:32.947279 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:28:32.947243    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:28:33.840821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.840777    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:28:33.841958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.841930    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e4d480b1-e162-4ad8-8aca-1c51def2da96" containerName="payload-processing"
Oct 31 16:28:33.841958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.841955    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d480b1-e162-4ad8-8aca-1c51def2da96" containerName="payload-processing"
Oct 31 16:28:33.842129 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.841977    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" containerName="payload-pre-processing"
Oct 31 16:28:33.842129 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.841985    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" containerName="payload-pre-processing"
Oct 31 16:28:33.842195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.842182    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1483bf0c-7ece-4838-9c45-3c936a6c0cd9" containerName="payload-pre-processing"
Oct 31 16:28:33.842228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.842208    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d480b1-e162-4ad8-8aca-1c51def2da96" containerName="payload-processing"
Oct 31 16:28:33.855964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.855930    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.931725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.931672    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.932079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.932055    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.932508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.932473    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.932754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.932733    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.932912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.932895    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.933095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.933065    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788g7\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.933565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.933534    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.933989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.933966    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.934157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.934138    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:33.977649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:33.964750    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a1b30a15-openshift-default-dockercfg-cmlh7\""
Oct 31 16:28:34.111851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.111753    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:28:34.160642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.159225    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.164653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.161191    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.164653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.159318    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.167865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.162703    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.168480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.168453    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.204654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.178875    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.205081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.205047    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.205373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.205357    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.222650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.209836    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.222650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.209884    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.222650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.209953    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788g7\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.222650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.190237    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.232983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.232947    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.245656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.235811    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.300935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.290929    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788g7\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.328255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.301610    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.375714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.347058    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.412112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.408728    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.564131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.564092    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:34.830875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.830840    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:28:34.931297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:34.931195    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:28:35.076708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.076315    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:28:35.082499 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:28:35.082472    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3b69c1_72fb_45f5_81f3_4cceef64c2f8.slice/crio-98178f1c2d6c352882b07827035b8596ed468582c4ac02c63da112233b2eff77 WatchSource:0}: Error finding container 98178f1c2d6c352882b07827035b8596ed468582c4ac02c63da112233b2eff77: Status 404 returned error can't find the container with id 98178f1c2d6c352882b07827035b8596ed468582c4ac02c63da112233b2eff77
Oct 31 16:28:35.085642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.085594    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:35.085729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.085673    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:35.085729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.085704    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:28:35.581058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.581023    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerStarted","Data":"4dca61d2db839764a81fbdc2d93aca2dd32309850535195b121f227d0c1a2817"}
Oct 31 16:28:35.581230 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.581074    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerStarted","Data":"98178f1c2d6c352882b07827035b8596ed468582c4ac02c63da112233b2eff77"}
Oct 31 16:28:35.694297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:35.694244    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podStartSLOduration=2.694230596 podStartE2EDuration="2.694230596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:35.688751397 +0000 UTC m=+1560.750326583" watchObservedRunningTime="2026-10-31 16:28:35.694230596 +0000 UTC m=+1560.755805782"
Oct 31 16:28:36.570640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:36.570097    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:36.583903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:36.583873    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:36.874992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:36.874877    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:36.879395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:36.879365    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:28:39.831725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:39.831662    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" podUID="06ebaca5-81af-48b7-b77a-1e313fb10505" containerName="payload-processing" containerID="cri-o://78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505" gracePeriod=25
Oct 31 16:28:39.932421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:39.932380    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" podUID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" containerName="payload-pre-processing" containerID="cri-o://7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4" gracePeriod=25
Oct 31 16:28:40.140817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.140782    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:28:40.231056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.231027    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:28:40.234360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.234329    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\") pod \"06ebaca5-81af-48b7-b77a-1e313fb10505\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") "
Oct 31 16:28:40.234557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.234537    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\") pod \"06ebaca5-81af-48b7-b77a-1e313fb10505\" (UID: \"06ebaca5-81af-48b7-b77a-1e313fb10505\") "
Oct 31 16:28:40.235374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.235079    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume" pod "06ebaca5-81af-48b7-b77a-1e313fb10505" (UID: "06ebaca5-81af-48b7-b77a-1e313fb10505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:40.235374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.235300    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ebaca5-81af-48b7-b77a-1e313fb10505-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:40.237248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.237217    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp" pod "06ebaca5-81af-48b7-b77a-1e313fb10505" (UID: "06ebaca5-81af-48b7-b77a-1e313fb10505"). InnerVolumeSpecName "kube-api-access-4g7gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:40.337829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.336470    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\" (UniqueName: \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\") pod \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") "
Oct 31 16:28:40.337829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.336795    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\") pod \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\" (UID: \"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4\") "
Oct 31 16:28:40.337829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.337557    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7gp\" (UniqueName: \"kubernetes.io/projected/06ebaca5-81af-48b7-b77a-1e313fb10505-kube-api-access-4g7gp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:40.338641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.338311    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume" pod "ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" (UID: "ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:28:40.342665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.341153    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f" pod "ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" (UID: "ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4"). InnerVolumeSpecName "kube-api-access-wsz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:28:40.439289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.439188    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:40.439289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.439222    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsz2f\" (UniqueName: \"kubernetes.io/projected/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4-kube-api-access-wsz2f\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:28:40.937137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.937102    4417 generic.go:356] "Generic (PLEG): container finished" podID="06ebaca5-81af-48b7-b77a-1e313fb10505" containerID="78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505" exitCode=0
Oct 31 16:28:40.937887 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.937857    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" event={"ID":"06ebaca5-81af-48b7-b77a-1e313fb10505","Type":"ContainerDied","Data":"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"}
Oct 31 16:28:40.938125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.938106    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh" event={"ID":"06ebaca5-81af-48b7-b77a-1e313fb10505","Type":"ContainerDied","Data":"f7fb4de65feb3a290d7f34578a056cdabcdabc9effb73d496a891778d0a636cb"}
Oct 31 16:28:40.938238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.938226    4417 scope.go:122] "RemoveContainer" containerID="78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"
Oct 31 16:28:40.938488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.938473    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"
Oct 31 16:28:40.946750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.946247    4417 generic.go:356] "Generic (PLEG): container finished" podID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" containerID="7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4" exitCode=0
Oct 31 16:28:40.946750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.946398    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" event={"ID":"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4","Type":"ContainerDied","Data":"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"}
Oct 31 16:28:40.946750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.946448    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q" event={"ID":"ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4","Type":"ContainerDied","Data":"2f63d9d2b119f095d1075abb4750905bcddbd891e4db593f7170ccc7eb6d5a71"}
Oct 31 16:28:40.946750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.946547    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"
Oct 31 16:28:40.969570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.969549    4417 scope.go:122] "RemoveContainer" containerID="78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"
Oct 31 16:28:40.970023 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:28:40.969999    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505\": container with ID starting with 78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505 not found: ID does not exist" containerID="78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"
Oct 31 16:28:40.970136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.970034    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505"} err="failed to get container status \"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505\": rpc error: code = NotFound desc = could not find container \"78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505\": container with ID starting with 78c8595e90935a84453d73c12143dd31f8d3839ac1dd72319144376550d7c505 not found: ID does not exist"
Oct 31 16:28:40.970136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.970061    4417 scope.go:122] "RemoveContainer" containerID="7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"
Oct 31 16:28:40.981080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.980947    4417 scope.go:122] "RemoveContainer" containerID="7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"
Oct 31 16:28:40.981465 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:28:40.981439    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4\": container with ID starting with 7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4 not found: ID does not exist" containerID="7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"
Oct 31 16:28:40.981567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:40.981476    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4"} err="failed to get container status \"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4\": rpc error: code = NotFound desc = could not find container \"7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4\": container with ID starting with 7ed9271ec86912cd3c2aab0faf0db1bb9d9f6dba246a9994a271f157702b3df4 not found: ID does not exist"
Oct 31 16:28:41.096435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:41.096386    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:28:41.100987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:41.100952    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ce4664d0-76b44c4f9f-8xs6q"]
Oct 31 16:28:41.201204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:41.201105    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:28:41.205073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:41.205046    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ce4664d0-56c8dc8f55-k7bzh"]
Oct 31 16:28:43.093810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:43.093735    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ebaca5-81af-48b7-b77a-1e313fb10505" path="/var/lib/kubelet/pods/06ebaca5-81af-48b7-b77a-1e313fb10505/volumes"
Oct 31 16:28:43.095528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:43.095501    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" path="/var/lib/kubelet/pods/ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4/volumes"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.060381    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069470    4417 cpu_manager.go:414] "containerMap: removing container" podUID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" containerName="payload-pre-processing"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069495    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" containerName="payload-pre-processing"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069546    4417 cpu_manager.go:414] "containerMap: removing container" podUID="06ebaca5-81af-48b7-b77a-1e313fb10505" containerName="payload-processing"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069556    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ebaca5-81af-48b7-b77a-1e313fb10505" containerName="payload-processing"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069772    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee53f8ce-12e6-45c6-bb5e-e8bdf0aa54f4" containerName="payload-pre-processing"
Oct 31 16:28:46.070684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.069794    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ebaca5-81af-48b7-b77a-1e313fb10505" containerName="payload-processing"
Oct 31 16:28:46.080643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.079210    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.119644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.119072    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:28:46.154474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.151892    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a1b30a15\""
Oct 31 16:28:46.154474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.152052    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a1b30a15-dockercfg-29vpd\""
Oct 31 16:28:46.222444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.204853    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.222444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.205214    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687bh\" (UniqueName: \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.371729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.364460    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.371729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.364753    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-687bh\" (UniqueName: \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.371729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.366175    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.504660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.450369    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:28:46.552776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.534917    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.593677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.592199    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:28:46.607941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.607505    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.607941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.607722    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtd9b\" (UniqueName: \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.657779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.610330    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-687bh\" (UniqueName: \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.709540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.709501    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.709771 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.709727    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtd9b\" (UniqueName: \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.710457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.710432    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.760272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.760235    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:46.762953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.762916    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtd9b\" (UniqueName: \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:46.924978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:46.924867    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:28:47.139514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:47.139482    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:28:47.140242 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:28:47.140213    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3827b8_a2ce_42af_aa8a_b4b5d9e3909e.slice/crio-77c21f68c6fe598e144e3eae78d71016d31952048afd5781063bd151b45ad69d WatchSource:0}: Error finding container 77c21f68c6fe598e144e3eae78d71016d31952048afd5781063bd151b45ad69d: Status 404 returned error can't find the container with id 77c21f68c6fe598e144e3eae78d71016d31952048afd5781063bd151b45ad69d
Oct 31 16:28:47.307296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:47.307188    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:28:47.308572 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:28:47.308543    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae53aae_528c_451c_a9cd_d0f41955daeb.slice/crio-15ecaab91763aa5072d9b2c1521b0f1699fe0566033b296cda10caaa73edc2ee WatchSource:0}: Error finding container 15ecaab91763aa5072d9b2c1521b0f1699fe0566033b296cda10caaa73edc2ee: Status 404 returned error can't find the container with id 15ecaab91763aa5072d9b2c1521b0f1699fe0566033b296cda10caaa73edc2ee
Oct 31 16:28:48.049492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.049455    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" event={"ID":"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e","Type":"ContainerStarted","Data":"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"}
Oct 31 16:28:48.051077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.051050    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" event={"ID":"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e","Type":"ContainerStarted","Data":"77c21f68c6fe598e144e3eae78d71016d31952048afd5781063bd151b45ad69d"}
Oct 31 16:28:48.051375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.051353    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:28:48.051598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.051574    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" event={"ID":"8ae53aae-528c-451c-a9cd-d0f41955daeb","Type":"ContainerStarted","Data":"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"}
Oct 31 16:28:48.051718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.051641    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" event={"ID":"8ae53aae-528c-451c-a9cd-d0f41955daeb","Type":"ContainerStarted","Data":"15ecaab91763aa5072d9b2c1521b0f1699fe0566033b296cda10caaa73edc2ee"}
Oct 31 16:28:48.157341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.157286    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" podStartSLOduration=3.1572728310000002 podStartE2EDuration="3.157272831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:48.154087765 +0000 UTC m=+1573.215662952" watchObservedRunningTime="2026-10-31 16:28:48.157272831 +0000 UTC m=+1573.218848016"
Oct 31 16:28:48.261683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:48.261612    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" podStartSLOduration=2.261599004 podStartE2EDuration="2.261599004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:48.257611512 +0000 UTC m=+1573.319186698" watchObservedRunningTime="2026-10-31 16:28:48.261599004 +0000 UTC m=+1573.323174188"
Oct 31 16:28:49.062237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:28:49.062203    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:29:00.083350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:00.083319    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:29:00.083810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:00.083437    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:29:02.954480 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:02.954434    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:29:04.339919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.339879    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:29:04.340420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.340136    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" podUID="e0176593-9391-4023-81cc-0c406b5ec2d7" containerName="maas-api" containerID="cri-o://022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2" gracePeriod=30
Oct 31 16:29:04.823462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.823429    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:29:04.954895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.938729    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\") pod \"e0176593-9391-4023-81cc-0c406b5ec2d7\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") "
Oct 31 16:29:04.954895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.938830    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\") pod \"e0176593-9391-4023-81cc-0c406b5ec2d7\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") "
Oct 31 16:29:04.954895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.938913    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") pod \"e0176593-9391-4023-81cc-0c406b5ec2d7\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") "
Oct 31 16:29:04.954895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.939038    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\" (UniqueName: \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\") pod \"e0176593-9391-4023-81cc-0c406b5ec2d7\" (UID: \"e0176593-9391-4023-81cc-0c406b5ec2d7\") "
Oct 31 16:29:04.956087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.956048    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca" pod "e0176593-9391-4023-81cc-0c406b5ec2d7" (UID: "e0176593-9391-4023-81cc-0c406b5ec2d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:04.961162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:04.961109    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9" pod "e0176593-9391-4023-81cc-0c406b5ec2d7" (UID: "e0176593-9391-4023-81cc-0c406b5ec2d7"). InnerVolumeSpecName "kube-api-access-pgkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:05.019878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.019750    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs" pod "e0176593-9391-4023-81cc-0c406b5ec2d7" (UID: "e0176593-9391-4023-81cc-0c406b5ec2d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:05.022671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.020100    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls" pod "e0176593-9391-4023-81cc-0c406b5ec2d7" (UID: "e0176593-9391-4023-81cc-0c406b5ec2d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:05.063981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.060061    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:05.063981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.060189    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0176593-9391-4023-81cc-0c406b5ec2d7-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:05.063981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.060205    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0176593-9391-4023-81cc-0c406b5ec2d7-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:05.063981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.060219    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgkb9\" (UniqueName: \"kubernetes.io/projected/e0176593-9391-4023-81cc-0c406b5ec2d7-kube-api-access-pgkb9\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:05.362653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.361991    4417 generic.go:356] "Generic (PLEG): container finished" podID="e0176593-9391-4023-81cc-0c406b5ec2d7" containerID="022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2" exitCode=0
Oct 31 16:29:05.362653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.362161    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" event={"ID":"e0176593-9391-4023-81cc-0c406b5ec2d7","Type":"ContainerDied","Data":"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"}
Oct 31 16:29:05.362653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.362216    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh" event={"ID":"e0176593-9391-4023-81cc-0c406b5ec2d7","Type":"ContainerDied","Data":"59b63b69deb116c23da655a3770f69652e1e4b9deb9faa379d816c34ddce9b8f"}
Oct 31 16:29:05.362653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.362247    4417 scope.go:122] "RemoveContainer" containerID="022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"
Oct 31 16:29:05.362653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.362423    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"
Oct 31 16:29:05.444695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.441835    4417 scope.go:122] "RemoveContainer" containerID="022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"
Oct 31 16:29:05.463647 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:05.460782    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2\": container with ID starting with 022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2 not found: ID does not exist" containerID="022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"
Oct 31 16:29:05.463647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.460841    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2"} err="failed to get container status \"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2\": rpc error: code = NotFound desc = could not find container \"022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2\": container with ID starting with 022bfa120182f7a62a39a8ba7617b39b03651532206545fe24b87bf750a35fd2 not found: ID does not exist"
Oct 31 16:29:05.536482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.534390    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:29:05.562640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:05.548934    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1bb9544-574f5bdd7-g8kfh"]
Oct 31 16:29:07.051710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:07.051673    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0176593-9391-4023-81cc-0c406b5ec2d7" path="/var/lib/kubelet/pods/e0176593-9391-4023-81cc-0c406b5ec2d7/volumes"
Oct 31 16:29:10.294455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.294415    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:29:10.295236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.295216    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e0176593-9391-4023-81cc-0c406b5ec2d7" containerName="maas-api"
Oct 31 16:29:10.295236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.295235    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0176593-9391-4023-81cc-0c406b5ec2d7" containerName="maas-api"
Oct 31 16:29:10.295395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.295357    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0176593-9391-4023-81cc-0c406b5ec2d7" containerName="maas-api"
Oct 31 16:29:10.300427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.300405    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.348740 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.348709    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6b1cca-openshift-default-dockercfg-lzzdh\""
Oct 31 16:29:10.400279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.400241    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:29:10.420990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.420953    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wz4\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421167    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421218    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421260    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421322    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421360    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421402    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421500    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.421581 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.421543    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.560697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.556679    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.560697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560111    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.560697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560366    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wz4\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.560697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560573    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.560697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560647    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.561091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560699    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.561091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.560802    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.561191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.561088    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.561191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.561173    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.571782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.562445    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.571782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.563934    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.571782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.564306    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.571782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.565284    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.574901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.574852    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.595638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.590826    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.616656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.612306    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.622150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.622080    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wz4\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.652768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.652685    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\") pod \"e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") " pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.923690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.917047    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:10.937717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.928286    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:29:10.937717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:10.928564    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerName="maas-api" containerID="cri-o://9f352d76c4b58a9d181b9f6b84ee056d1f75be2b9997ca5b8faac528968c9b90" gracePeriod=30
Oct 31 16:29:11.694079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:11.694009    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.103:8443/health\": dial tcp 10.133.0.103:8443: connect: connection refused"
Oct 31 16:29:11.706540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:11.706504    4417 generic.go:356] "Generic (PLEG): container finished" podID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerID="9f352d76c4b58a9d181b9f6b84ee056d1f75be2b9997ca5b8faac528968c9b90" exitCode=0
Oct 31 16:29:11.706930 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:11.706886    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" event={"ID":"e06c87d2-39aa-484b-b7cc-708e915319f6","Type":"ContainerDied","Data":"9f352d76c4b58a9d181b9f6b84ee056d1f75be2b9997ca5b8faac528968c9b90"}
Oct 31 16:29:12.022646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.021845    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:29:12.047695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.045494    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:29:12.104517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.103203    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:12.104517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.103304    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:12.104517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.103349    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:12.170967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.170934    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\" (UniqueName: \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\") pod \"e06c87d2-39aa-484b-b7cc-708e915319f6\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") "
Oct 31 16:29:12.171299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.171279    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\") pod \"e06c87d2-39aa-484b-b7cc-708e915319f6\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") "
Oct 31 16:29:12.171965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.171939    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\") pod \"e06c87d2-39aa-484b-b7cc-708e915319f6\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") "
Oct 31 16:29:12.172379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.172356    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") pod \"e06c87d2-39aa-484b-b7cc-708e915319f6\" (UID: \"e06c87d2-39aa-484b-b7cc-708e915319f6\") "
Oct 31 16:29:12.174992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.174967    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca" pod "e06c87d2-39aa-484b-b7cc-708e915319f6" (UID: "e06c87d2-39aa-484b-b7cc-708e915319f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:12.199368 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.199307    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m" pod "e06c87d2-39aa-484b-b7cc-708e915319f6" (UID: "e06c87d2-39aa-484b-b7cc-708e915319f6"). InnerVolumeSpecName "kube-api-access-jtq4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:12.220264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.220227    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs" pod "e06c87d2-39aa-484b-b7cc-708e915319f6" (UID: "e06c87d2-39aa-484b-b7cc-708e915319f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:12.220978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.220948    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls" pod "e06c87d2-39aa-484b-b7cc-708e915319f6" (UID: "e06c87d2-39aa-484b-b7cc-708e915319f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:12.300650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.288420    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:12.300650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.288461    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e06c87d2-39aa-484b-b7cc-708e915319f6-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:12.300650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.288476    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtq4m\" (UniqueName: \"kubernetes.io/projected/e06c87d2-39aa-484b-b7cc-708e915319f6-kube-api-access-jtq4m\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:12.300650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.288491    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e06c87d2-39aa-484b-b7cc-708e915319f6-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:12.746843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.746789    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerStarted","Data":"c64f3b38258e4595a8f630e3d86197e7080f3fe4c6b27744a089ca20f5dc8a0b"}
Oct 31 16:29:12.747354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.746868    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerStarted","Data":"ef6f15402a6da9673fd006836bbd695471b9b4a91639840a705d3b886c0f0ea6"}
Oct 31 16:29:12.749896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.749872    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"
Oct 31 16:29:12.750033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.749923    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt" event={"ID":"e06c87d2-39aa-484b-b7cc-708e915319f6","Type":"ContainerDied","Data":"e7c046c8cbb2ff5062efd497297c6c9cd6360a0efeb62064afbd5add5b52805b"}
Oct 31 16:29:12.750033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.749998    4417 scope.go:122] "RemoveContainer" containerID="9f352d76c4b58a9d181b9f6b84ee056d1f75be2b9997ca5b8faac528968c9b90"
Oct 31 16:29:12.856593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.856471    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podStartSLOduration=2.856455908 podStartE2EDuration="2.856455908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:12.855361584 +0000 UTC m=+1597.916936771" watchObservedRunningTime="2026-10-31 16:29:12.856455908 +0000 UTC m=+1597.918031093"
Oct 31 16:29:12.920859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.920826    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:12.967718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.967683    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:29:12.972989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:12.972959    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ce4664d0-5897dbd45-xgngt"]
Oct 31 16:29:13.129509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:13.129411    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" path="/var/lib/kubelet/pods/e06c87d2-39aa-484b-b7cc-708e915319f6/volumes"
Oct 31 16:29:13.851011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:13.850968    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:29:13.851555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:13.851250    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy" containerID="cri-o://fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7" gracePeriod=30
Oct 31 16:29:14.082649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:14.076589    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:14.764492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:14.764457    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:14.765424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:14.765403    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:29:15.565757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.565704    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:29:15.567365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.567337    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerName="maas-api"
Oct 31 16:29:15.567559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.567544    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerName="maas-api"
Oct 31 16:29:15.567924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.567907    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06c87d2-39aa-484b-b7cc-708e915319f6" containerName="maas-api"
Oct 31 16:29:15.573917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.573894    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.624660    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-822dd4-openshift-default-dockercfg-qvpfr\""
Oct 31 16:29:15.657877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.657835    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658112    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658165    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nz4\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658245    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658324    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658431    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658529    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658683    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.658799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.658717    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.675462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.675429    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:29:15.788722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.762908    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.794579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.794535    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.794947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.794920    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.808640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.795147    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.808949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.808913    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nz4\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.809204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.809183    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.809404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.809384    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.809681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.809660    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.826683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.809906    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.826994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.815536    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.827384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.815794    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.827536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.816035    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.827696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.816035    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.827844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.816706    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.855871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.855828    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.883662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.871356    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.910514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.910473    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nz4\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:15.923642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:15.923540    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\") pod \"e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") " pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:16.207037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:16.206945    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:17.549439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.549400    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:29:17.585726 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:17.585695    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241b81b1_5aa3_4bd4_8e17_f857aceb4a87.slice/crio-c1e0d426b9bd9c5baed0401be4511e3101af85149c392160450f21cd8d28297f WatchSource:0}: Error finding container c1e0d426b9bd9c5baed0401be4511e3101af85149c392160450f21cd8d28297f: Status 404 returned error can't find the container with id c1e0d426b9bd9c5baed0401be4511e3101af85149c392160450f21cd8d28297f
Oct 31 16:29:17.589038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.589003    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.589131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.589071    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.589131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.589105    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:17.922322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.922227    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerStarted","Data":"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"}
Oct 31 16:29:17.922482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:17.922331    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerStarted","Data":"c1e0d426b9bd9c5baed0401be4511e3101af85149c392160450f21cd8d28297f"}
Oct 31 16:29:18.108640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:18.096819    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podStartSLOduration=3.0967976 podStartE2EDuration="3.0967976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:18.054573516 +0000 UTC m=+1603.116148704" watchObservedRunningTime="2026-10-31 16:29:18.0967976 +0000 UTC m=+1603.158372785"
Oct 31 16:29:18.208083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:18.208004    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:18.211475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:18.211448    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:29:18.211703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:18.211679    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:29:19.152828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.152797    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:29:19.213454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.213419    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:19.299177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299142    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299259    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299307    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299365 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299354    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299425    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299505    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299555    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299548    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:19.299819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299753    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299717    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:19.299819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299782    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\") pod \"cb536711-6e8c-4881-98a2-6c379a48f3a1\" (UID: \"cb536711-6e8c-4881-98a2-6c379a48f3a1\") "
Oct 31 16:29:19.299819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299796    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:19.300022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.299843    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:19.300084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300023    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:19.300605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300579    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.300746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300610    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.300746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300660    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb536711-6e8c-4881-98a2-6c379a48f3a1-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.300746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300676    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.300746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.300690    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.302076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.302051    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:19.302173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.302121    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:19.302223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.302176    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "kube-api-access-4j9vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:19.302223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.302197    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy" pod "cb536711-6e8c-4881-98a2-6c379a48f3a1" (UID: "cb536711-6e8c-4881-98a2-6c379a48f3a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:19.402563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.402476    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.402563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.402506    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.402563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.402519    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9vq\" (UniqueName: \"kubernetes.io/projected/cb536711-6e8c-4881-98a2-6c379a48f3a1-kube-api-access-4j9vq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.402563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.402529    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb536711-6e8c-4881-98a2-6c379a48f3a1-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:19.893339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.893290    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:29:19.894671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.894640    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:19.894847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.894831    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:19.895276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.895256    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:19.895431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.895416    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:19.904637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.903770    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:19.959640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.952567    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:29:19.994660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.967684    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6b1cca\""
Oct 31 16:29:19.994660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:19.967910    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6b1cca-dockercfg-24f54\""
Oct 31 16:29:20.044644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.040447    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.044644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.040801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzms\" (UniqueName: \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.137502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.137453    4417 generic.go:356] "Generic (PLEG): container finished" podID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerID="fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7" exitCode=0
Oct 31 16:29:20.138561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.138528    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"
Oct 31 16:29:20.143190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.143135    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerDied","Data":"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"}
Oct 31 16:29:20.143336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.143215    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw" event={"ID":"cb536711-6e8c-4881-98a2-6c379a48f3a1","Type":"ContainerDied","Data":"278e8a47e3c52163fe47a704d5231705ef9802423f88f1c3195b0c346671dbe3"}
Oct 31 16:29:20.143336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.143245    4417 scope.go:122] "RemoveContainer" containerID="fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"
Oct 31 16:29:20.154015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.153949    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:20.157455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.156456    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzms\" (UniqueName: \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.157455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.156864    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.159078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.159053    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.163779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.162158    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:29:20.185196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.181713    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:29:20.185196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.182887    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:20.185196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.182912    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" containerName="istio-proxy"
Oct 31 16:29:20.190531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.189321    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.286339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.286273    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:29:20.374560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.374525    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzms\" (UniqueName: \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\") pod \"payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.375468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.375443    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:20.395232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.395148    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.395471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.395378    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77xw\" (UniqueName: \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.440871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.440378    4417 scope.go:122] "RemoveContainer" containerID="b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"
Oct 31 16:29:20.530208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.530167    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77xw\" (UniqueName: \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.530804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.530767    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.533447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.533427    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.588220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.588067    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77xw\" (UniqueName: \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\") pod \"payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:20.678938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.676238    4417 scope.go:122] "RemoveContainer" containerID="fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"
Oct 31 16:29:20.754643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.738589    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:29:20.754643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.746767    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ce4664d0-openshift-default-5c55fff65c-sn7tw"]
Oct 31 16:29:20.766649 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:20.766584    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7\": container with ID starting with fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7 not found: ID does not exist" containerID="fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"
Oct 31 16:29:20.766845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.766653    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7"} err="failed to get container status \"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7\": rpc error: code = NotFound desc = could not find container \"fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7\": container with ID starting with fe3f8f65aef7726134e410eb6c6b67f29e3b099ec19e15da72ea9e603058c3f7 not found: ID does not exist"
Oct 31 16:29:20.766845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.766683    4417 scope.go:122] "RemoveContainer" containerID="b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"
Oct 31 16:29:20.774639 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:20.767140    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb\": container with ID starting with b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb not found: ID does not exist" containerID="b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"
Oct 31 16:29:20.774639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.767181    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb"} err="failed to get container status \"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb\": rpc error: code = NotFound desc = could not find container \"b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb\": container with ID starting with b9a56c933151b217b9b02df5786f5bdd1e5d299906f692f8f38c44df146449fb not found: ID does not exist"
Oct 31 16:29:20.833446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:20.833407    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:21.053835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.053802    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb536711-6e8c-4881-98a2-6c379a48f3a1" path="/var/lib/kubelet/pods/cb536711-6e8c-4881-98a2-6c379a48f3a1/volumes"
Oct 31 16:29:21.109706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.109673    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:29:21.111594 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:21.111563    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bbbab1_405f_4534_bce0_76ab238a4569.slice/crio-61a761047286c3bf8c180159eaaace3112e2453b6dac643d689be2ad91ec791c WatchSource:0}: Error finding container 61a761047286c3bf8c180159eaaace3112e2453b6dac643d689be2ad91ec791c: Status 404 returned error can't find the container with id 61a761047286c3bf8c180159eaaace3112e2453b6dac643d689be2ad91ec791c
Oct 31 16:29:21.148024 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.147994    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" event={"ID":"b4bbbab1-405f-4534-bce0-76ab238a4569","Type":"ContainerStarted","Data":"61a761047286c3bf8c180159eaaace3112e2453b6dac643d689be2ad91ec791c"}
Oct 31 16:29:21.215710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.215682    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:29:21.216133 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:21.216108    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f57064_04ed_4fb1_98be_d49e9911384c.slice/crio-3d5e66c80431e5ad73e8ecda253bd76c34c7478efc68c44c2734d28f4bc7761e WatchSource:0}: Error finding container 3d5e66c80431e5ad73e8ecda253bd76c34c7478efc68c44c2734d28f4bc7761e: Status 404 returned error can't find the container with id 3d5e66c80431e5ad73e8ecda253bd76c34c7478efc68c44c2734d28f4bc7761e
Oct 31 16:29:21.897096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.897057    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:29:21.897359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.897319    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" containerID="cri-o://12b0f14580a0bcf2d2b541c6dc3b1aea4fcc7f6674d8e5dc1c2271c0fdaf9e92" gracePeriod=30
Oct 31 16:29:21.943886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.943848    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:29:21.944171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.944122    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" containerID="cri-o://ae6b7a60d2391780d3f8738654ae7bd194e362c46019cd634000465a2f18f138" gracePeriod=30
Oct 31 16:29:21.951009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.950969    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:29:21.951241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.951217    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" containerID="cri-o://1ce30bdf30b54ff325acdc7c06ff214378f7802f10db68cc4b890598e324757f" gracePeriod=30
Oct 31 16:29:21.960573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.960544    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:29:21.960771 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.960748    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy" containerID="cri-o://a762102448a8a0e16610cad924b0e6202a3b8acef27c234df841e760cff3db0a" gracePeriod=30
Oct 31 16:29:21.991092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.991055    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:29:21.991482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.991310    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" containerID="cri-o://50d76eafa7a1a762b80e10038acd1cf10ff44c51b5d7d3d024d5f0ed1747c710" gracePeriod=30
Oct 31 16:29:22.005072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.996725    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:29:22.005072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:21.998604    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy" containerID="cri-o://2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627" gracePeriod=30
Oct 31 16:29:22.006698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.006666    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:29:22.007044 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.006901    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy" containerID="cri-o://47d5d7ad276c262fadd8bea255a0bb581212381cebf279903b41fedc00d229ae" gracePeriod=30
Oct 31 16:29:22.015450 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.013180    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:29:22.018169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.018138    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy" containerID="cri-o://1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3" gracePeriod=30
Oct 31 16:29:22.044637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.044187    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:29:22.044637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.044448    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy" containerID="cri-o://001eb2c700e867dac46f77d9ff90902fc47b9b27eeddb1751f195254e17510ae" gracePeriod=30
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.051980    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.052264    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy" containerID="cri-o://0f88b900f3de183f042cfee56b489317b3f4506f4935bbeba4c63eea840697a2" gracePeriod=30
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.057993    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.058208    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy" containerID="cri-o://574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad" gracePeriod=30
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.069668    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:29:22.070259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.069909    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" containerID="cri-o://01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a" gracePeriod=30
Oct 31 16:29:22.160723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.156949    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" event={"ID":"93f57064-04ed-4fb1-98be-d49e9911384c","Type":"ContainerStarted","Data":"215e0ab0b759ddead6736161f80f497699780e0f2c92c50edb1d938eb025f95c"}
Oct 31 16:29:22.160723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.157018    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" event={"ID":"93f57064-04ed-4fb1-98be-d49e9911384c","Type":"ContainerStarted","Data":"3d5e66c80431e5ad73e8ecda253bd76c34c7478efc68c44c2734d28f4bc7761e"}
Oct 31 16:29:22.160723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.157048    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:22.160723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.159386    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" event={"ID":"b4bbbab1-405f-4534-bce0-76ab238a4569","Type":"ContainerStarted","Data":"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"}
Oct 31 16:29:22.160723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:22.159475    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:23.070027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:23.062186    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" podStartSLOduration=4.062165581 podStartE2EDuration="4.062165581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:23.00477394 +0000 UTC m=+1608.066349144" watchObservedRunningTime="2026-10-31 16:29:23.062165581 +0000 UTC m=+1608.123740768"
Oct 31 16:29:23.515714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:23.515579    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" podStartSLOduration=4.51556485 podStartE2EDuration="4.51556485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:23.512563325 +0000 UTC m=+1608.574138509" watchObservedRunningTime="2026-10-31 16:29:23.51556485 +0000 UTC m=+1608.577140029"
Oct 31 16:29:24.029477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:24.029438    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:29:24.029692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:24.029512    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:29:26.721639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.704051    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:29:26.721639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.704108    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:29:26.812640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.807655    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:29:26.816638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.813270    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:26.862648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.862445    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-822dd4-dockercfg-9hchx\""
Oct 31 16:29:26.918122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.914843    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-822dd4\""
Oct 31 16:29:26.918122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.916185    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8q7p\" (UniqueName: \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:26.918122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:26.916525    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.281098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.201423    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.281098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.201698    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8q7p\" (UniqueName: \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.281098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.203536    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.281098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.278249    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8q7p\" (UniqueName: \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\") pod \"payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.559186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.557965    4417 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 16:29:27.559186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.558066    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 16:29:27.581447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.581411    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:27.686117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.683508    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:29:27.686117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.683601    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:29:27.723753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.723028    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:29:27.723753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.723205    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.742817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.742215    4417 generic.go:356] "Generic (PLEG): container finished" podID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerID="47d5d7ad276c262fadd8bea255a0bb581212381cebf279903b41fedc00d229ae" exitCode=0
Oct 31 16:29:27.742817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.742379    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" event={"ID":"90b0f869-a6fa-4bb5-82fc-eccf8a08f759","Type":"ContainerDied","Data":"47d5d7ad276c262fadd8bea255a0bb581212381cebf279903b41fedc00d229ae"}
Oct 31 16:29:27.767948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.767649    4417 generic.go:356] "Generic (PLEG): container finished" podID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerID="001eb2c700e867dac46f77d9ff90902fc47b9b27eeddb1751f195254e17510ae" exitCode=0
Oct 31 16:29:27.767948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.767742    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerDied","Data":"001eb2c700e867dac46f77d9ff90902fc47b9b27eeddb1751f195254e17510ae"}
Oct 31 16:29:27.767948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.767799    4417 scope.go:122] "RemoveContainer" containerID="e2c8dfb191f63b0e66c666dbcb4562a55efe3acb0dda8f755f85a038fb836c23"
Oct 31 16:29:27.804150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.801756    4417 generic.go:356] "Generic (PLEG): container finished" podID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerID="a762102448a8a0e16610cad924b0e6202a3b8acef27c234df841e760cff3db0a" exitCode=0
Oct 31 16:29:27.804150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.801876    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" event={"ID":"408aa05b-4a02-4177-ad74-ffa930bd8dc2","Type":"ContainerDied","Data":"a762102448a8a0e16610cad924b0e6202a3b8acef27c234df841e760cff3db0a"}
Oct 31 16:29:27.804545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.804366    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmbg\" (UniqueName: \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.804545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.804442    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.806213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.806005    4417 generic.go:356] "Generic (PLEG): container finished" podID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerID="12b0f14580a0bcf2d2b541c6dc3b1aea4fcc7f6674d8e5dc1c2271c0fdaf9e92" exitCode=0
Oct 31 16:29:27.806213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.806126    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" event={"ID":"def7b5ed-7f7a-48d2-b253-a5e06e0272ca","Type":"ContainerDied","Data":"12b0f14580a0bcf2d2b541c6dc3b1aea4fcc7f6674d8e5dc1c2271c0fdaf9e92"}
Oct 31 16:29:27.808680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.808486    4417 generic.go:356] "Generic (PLEG): container finished" podID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerID="0f88b900f3de183f042cfee56b489317b3f4506f4935bbeba4c63eea840697a2" exitCode=0
Oct 31 16:29:27.808940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.808862    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" event={"ID":"72cdad85-dd92-4e37-9562-2acd458fc1d8","Type":"ContainerDied","Data":"0f88b900f3de183f042cfee56b489317b3f4506f4935bbeba4c63eea840697a2"}
Oct 31 16:29:27.811390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.811187    4417 generic.go:356] "Generic (PLEG): container finished" podID="58203f05-1714-450a-a50c-42592d06eb5f" containerID="50d76eafa7a1a762b80e10038acd1cf10ff44c51b5d7d3d024d5f0ed1747c710" exitCode=0
Oct 31 16:29:27.811390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.811275    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" event={"ID":"58203f05-1714-450a-a50c-42592d06eb5f","Type":"ContainerDied","Data":"50d76eafa7a1a762b80e10038acd1cf10ff44c51b5d7d3d024d5f0ed1747c710"}
Oct 31 16:29:27.835872 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:27.826099    4417 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b607a_4354_403f_a966_fb91dc414bf7.slice/crio-conmon-1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebc57bf_e8b0_46e9_9c71_09cb0f8fcf74.slice/crio-01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebc57bf_e8b0_46e9_9c71_09cb0f8fcf74.slice/crio-conmon-01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80599a52_229a_4d25_a347_687580a623e8.slice/crio-2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead36ef_9851_4d9a_a53a_9df732c6303a.slice/crio-conmon-574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 16:29:27.848826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.846671    4417 generic.go:356] "Generic (PLEG): container finished" podID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerID="01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a" exitCode=0
Oct 31 16:29:27.848826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.846801    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" event={"ID":"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74","Type":"ContainerDied","Data":"01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a"}
Oct 31 16:29:27.861654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.857404    4417 generic.go:356] "Generic (PLEG): container finished" podID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerID="ae6b7a60d2391780d3f8738654ae7bd194e362c46019cd634000465a2f18f138" exitCode=0
Oct 31 16:29:27.861654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.857534    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerDied","Data":"ae6b7a60d2391780d3f8738654ae7bd194e362c46019cd634000465a2f18f138"}
Oct 31 16:29:27.861654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.857603    4417 scope.go:122] "RemoveContainer" containerID="6c0496446a5747a0565b44bce25c76af942f839c297f5949df9a20880c023923"
Oct 31 16:29:27.922770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.920307    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmbg\" (UniqueName: \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.922770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.920380    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.922770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.921888    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:27.923339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.923309    4417 generic.go:356] "Generic (PLEG): container finished" podID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerID="1ce30bdf30b54ff325acdc7c06ff214378f7802f10db68cc4b890598e324757f" exitCode=0
Oct 31 16:29:27.923475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.923402    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" event={"ID":"8ce2dce3-4cfd-476b-b812-1c0926b8b88a","Type":"ContainerDied","Data":"1ce30bdf30b54ff325acdc7c06ff214378f7802f10db68cc4b890598e324757f"}
Oct 31 16:29:27.976011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:27.975783    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmbg\" (UniqueName: \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\") pod \"payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:28.095829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.095801    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:29:28.147033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.145951    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:28.254417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254071    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254204    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254300    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254374    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254545    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254665    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254709    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.254787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254769    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.255006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254825    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.255006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.254866    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\") pod \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\" (UID: \"8ce2dce3-4cfd-476b-b812-1c0926b8b88a\") "
Oct 31 16:29:28.255800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.255597    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.257228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.256260    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.257228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.256430    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.258942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.258908    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.259046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.259016    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.260539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.260504    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "kube-api-access-wwlph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.260781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.260750    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.260998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.260970    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.261602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.261581    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token" pod "8ce2dce3-4cfd-476b-b812-1c0926b8b88a" (UID: "8ce2dce3-4cfd-476b-b812-1c0926b8b88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.279355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.279005    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:29:28.281733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.281490    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:29:28.284570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.284547    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:29:28.286728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.286708    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363099    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363164    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363189    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363237    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363305    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363388    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363415    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363451    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363490    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363544    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363572    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363634    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363689    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363719    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363770    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363794    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363841    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\") pod \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\" (UID: \"def7b5ed-7f7a-48d2-b253-a5e06e0272ca\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.363987    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\") pod \"393b607a-4354-403f-a966-fb91dc414bf7\" (UID: \"393b607a-4354-403f-a966-fb91dc414bf7\") "
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364828    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364847    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364862    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364874    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364885    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364898    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlph\" (UniqueName: \"kubernetes.io/projected/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-kube-api-access-wwlph\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364911    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.365120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.364923    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ce2dce3-4cfd-476b-b812-1c0926b8b88a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.367562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.367282    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.369342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.369077    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.370346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.370108    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.370957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.370711    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.371249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.371230    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.372114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.372097    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.372362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.372347    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.372592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.372578    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.388781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.387870    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.389791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.389751    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.391519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.391484    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.397031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.396954    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:29:28.428493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.428456    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.429161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.429124    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.429803 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.429766    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.430078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.430044    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "kube-api-access-kzsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.433487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.430856    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.433487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.433423    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy" pod "393b607a-4354-403f-a966-fb91dc414bf7" (UID: "393b607a-4354-403f-a966-fb91dc414bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.434117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.434095    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h" pod "def7b5ed-7f7a-48d2-b253-a5e06e0272ca" (UID: "def7b5ed-7f7a-48d2-b253-a5e06e0272ca"). InnerVolumeSpecName "kube-api-access-2mz2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.463225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.462889    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465578    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465669    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465732    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465808    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465830    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465859    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465921    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465947    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.465986    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466006    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466058    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466136    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466155    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466182    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466213    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466230    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466247    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\") pod \"58203f05-1714-450a-a50c-42592d06eb5f\" (UID: \"58203f05-1714-450a-a50c-42592d06eb5f\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466329    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\") pod \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\" (UID: \"90b0f869-a6fa-4bb5-82fc-eccf8a08f759\") "
Oct 31 16:29:28.466755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.466526    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467157    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467181    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467195    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467209    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467224    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467238    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mz2h\" (UniqueName: \"kubernetes.io/projected/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-kube-api-access-2mz2h\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467253    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393b607a-4354-403f-a966-fb91dc414bf7-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467267    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393b607a-4354-403f-a966-fb91dc414bf7-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467281    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467294    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467339    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467354    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzsvl\" (UniqueName: \"kubernetes.io/projected/393b607a-4354-403f-a966-fb91dc414bf7-kube-api-access-kzsvl\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467367    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467380    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467393    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467406    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467419    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467433    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/def7b5ed-7f7a-48d2-b253-a5e06e0272ca-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467446    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393b607a-4354-403f-a966-fb91dc414bf7-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.467868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.467640    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.468183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.468023    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.468578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.468381    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.468924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.468901    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.469768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.469501    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.470197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.469957    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.470803 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.470532    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.473240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.473042    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.473714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.473455    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.474741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.473853    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.474741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.473922    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.476987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.476956    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "kube-api-access-2b4sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.477354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.477254    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.477354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.477284    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.477354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.477314    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo" pod "90b0f869-a6fa-4bb5-82fc-eccf8a08f759" (UID: "90b0f869-a6fa-4bb5-82fc-eccf8a08f759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.477674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.477645    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "kube-api-access-bd96s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.477895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.477860    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token" pod "58203f05-1714-450a-a50c-42592d06eb5f" (UID: "58203f05-1714-450a-a50c-42592d06eb5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.482349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.482201    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:29:28.485872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.485659    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:29:28.490102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.489897    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:29:28.492999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.492771    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:29:28.496342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.496321    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:29:28.568497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568445    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.568706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568520    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.568706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568558    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.568706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568587    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.568706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568658    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.568706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568701    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568796    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568832    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568874    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568908    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568942    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.568975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568975    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.568998    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569023    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569058    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569099    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569147    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569150    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569193    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569242    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569289    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569320    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569413    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569457    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569485    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569524    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569604    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.569598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569671    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569701    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569752    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569837    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569864    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569900    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569951    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.569976    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570010    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570019    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570047    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570079    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\") pod \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\" (UID: \"408aa05b-4a02-4177-ad74-ffa930bd8dc2\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570109    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570165    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570246    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\") pod \"72cdad85-dd92-4e37-9562-2acd458fc1d8\" (UID: \"72cdad85-dd92-4e37-9562-2acd458fc1d8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570301    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570329    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570388    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570439    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\") pod \"80599a52-229a-4d25-a347-687580a623e8\" (UID: \"80599a52-229a-4d25-a347-687580a623e8\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570495    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\") pod \"dead36ef-9851-4d9a-a53a-9df732c6303a\" (UID: \"dead36ef-9851-4d9a-a53a-9df732c6303a\") "
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570502    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.570566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.570533    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\") pod \"1baba935-5cd0-44bd-a815-f5742357b8b6\" (UID: \"1baba935-5cd0-44bd-a815-f5742357b8b6\") "
Oct 31 16:29:28.571743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571339    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.571743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571567    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.571858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571804    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4sr\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-kube-api-access-2b4sr\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.571858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571824    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.571858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571839    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58203f05-1714-450a-a50c-42592d06eb5f-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.571858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571853    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571868    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571882    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571898    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571912    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571928    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571942    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571956    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571969    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571983    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571996    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58203f05-1714-450a-a50c-42592d06eb5f-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572009    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572022    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572034    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572048    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58203f05-1714-450a-a50c-42592d06eb5f-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572060    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572074    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd96s\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-kube-api-access-bd96s\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572089    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572103    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58203f05-1714-450a-a50c-42592d06eb5f-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572116    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b0f869-a6fa-4bb5-82fc-eccf8a08f759-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572131    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571849    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.571875    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572177    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572196    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572377    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.572600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572456    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.573158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572662    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.573158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572726    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.573158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.572897    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.574142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.573834    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.574142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.574023    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.575931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.575626    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.577905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.577880    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.578093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.578075    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.580008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.579980    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "kube-api-access-7pj2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.581357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.581159    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.581587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.581429    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.581587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.581466    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.581808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.581764    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.582635 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.582589    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.582822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.582796    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.583263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583050    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.583263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583109    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.583263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583170    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.583737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583533    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.583737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583546    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.583737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583597    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.583737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583627    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy" pod "dead36ef-9851-4d9a-a53a-9df732c6303a" (UID: "dead36ef-9851-4d9a-a53a-9df732c6303a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.583737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.583691    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "kube-api-access-6k6cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.584391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.584369    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "kube-api-access-q6sg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.586124 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.586098    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.586712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.586679    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.586892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.586859    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo" pod "80599a52-229a-4d25-a347-687580a623e8" (UID: "80599a52-229a-4d25-a347-687580a623e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.586892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.586869    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.587207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.587164    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj" pod "408aa05b-4a02-4177-ad74-ffa930bd8dc2" (UID: "408aa05b-4a02-4177-ad74-ffa930bd8dc2"). InnerVolumeSpecName "kube-api-access-vndbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.587291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.587270    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "kube-api-access-k6j4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.587685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.587666    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token" pod "1baba935-5cd0-44bd-a815-f5742357b8b6" (UID: "1baba935-5cd0-44bd-a815-f5742357b8b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.588564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.588541    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy" pod "72cdad85-dd92-4e37-9562-2acd458fc1d8" (UID: "72cdad85-dd92-4e37-9562-2acd458fc1d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.673013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.672971    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673034    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673075    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673118    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673296    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673342    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673378    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673414    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673536    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673567    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673631    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.673679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673668    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673706    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673754    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.673857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673827    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\") pod \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\" (UID: \"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74\") "
Oct 31 16:29:28.674007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673877    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.674007 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.673949    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\") pod \"7161f702-5ab0-4b78-9f49-00d10a01557d\" (UID: \"7161f702-5ab0-4b78-9f49-00d10a01557d\") "
Oct 31 16:29:28.674862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674835    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674865    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674880    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674893    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6j4n\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-kube-api-access-k6j4n\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674908    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80599a52-229a-4d25-a347-687580a623e8-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674921    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674936    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80599a52-229a-4d25-a347-687580a623e8-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674949    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674962    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.674984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674975    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndbj\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-kube-api-access-vndbj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.674990    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675006    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pj2l\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-kube-api-access-7pj2l\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675021    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675046    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675061    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6sg5\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-kube-api-access-q6sg5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675075    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675090    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675102    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675114    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675130    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dead36ef-9851-4d9a-a53a-9df732c6303a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675145    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675158    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cdad85-dd92-4e37-9562-2acd458fc1d8-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675162    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675171    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675217    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k6cm\" (UniqueName: \"kubernetes.io/projected/72cdad85-dd92-4e37-9562-2acd458fc1d8-kube-api-access-6k6cm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675237    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1baba935-5cd0-44bd-a815-f5742357b8b6-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675253    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675266    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675280    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dead36ef-9851-4d9a-a53a-9df732c6303a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675293    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1baba935-5cd0-44bd-a815-f5742357b8b6-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675306    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cdad85-dd92-4e37-9562-2acd458fc1d8-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675347    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675360    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dead36ef-9851-4d9a-a53a-9df732c6303a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675372    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1baba935-5cd0-44bd-a815-f5742357b8b6-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675384    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80599a52-229a-4d25-a347-687580a623e8-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675396    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675411    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675418    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408aa05b-4a02-4177-ad74-ffa930bd8dc2-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675456    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80599a52-229a-4d25-a347-687580a623e8-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675476    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408aa05b-4a02-4177-ad74-ffa930bd8dc2-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.675486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675481    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.676098    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.675994    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.676150    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.676171    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.676082    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.676342    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.677132    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.680383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.677373    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.681587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.681552    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.681712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.681647    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.682084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.682059    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.682259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.682233    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:29:28.684142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.684121    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p" pod "7161f702-5ab0-4b78-9f49-00d10a01557d" (UID: "7161f702-5ab0-4b78-9f49-00d10a01557d"). InnerVolumeSpecName "kube-api-access-qzs7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.684251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.684195    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "kube-api-access-jck66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:28.685368 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.685338    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo" pod "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" (UID: "8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:29:28.764512 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:28.764474    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515acf0c_2a58_44f1_92bc_ae872d52dd3a.slice/crio-2b60e76e0597d2e6339d66df1e7e2fe0fec9d7d07904a1a824d28b4b442ea300 WatchSource:0}: Error finding container 2b60e76e0597d2e6339d66df1e7e2fe0fec9d7d07904a1a824d28b4b442ea300: Status 404 returned error can't find the container with id 2b60e76e0597d2e6339d66df1e7e2fe0fec9d7d07904a1a824d28b4b442ea300
Oct 31 16:29:28.765182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.765145    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:29:28.777188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777164    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777188    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777197    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7161f702-5ab0-4b78-9f49-00d10a01557d-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777207    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777217    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777226    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777234    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzs7p\" (UniqueName: \"kubernetes.io/projected/7161f702-5ab0-4b78-9f49-00d10a01557d-kube-api-access-qzs7p\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777242    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777252    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777260    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777270    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777278    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777286    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777294    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jck66\" (UniqueName: \"kubernetes.io/projected/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-kube-api-access-jck66\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777302    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7161f702-5ab0-4b78-9f49-00d10a01557d-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777310    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777319    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.777357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.777327    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:28.894914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.894842    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:29:28.904940 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:28.904906    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de4c19b_ad77_407d_be89_6ac94284fe85.slice/crio-1ddd0b05f2a2fef483730c9f6f43af0469a62136553b2833000dab36001925b2 WatchSource:0}: Error finding container 1ddd0b05f2a2fef483730c9f6f43af0469a62136553b2833000dab36001925b2: Status 404 returned error can't find the container with id 1ddd0b05f2a2fef483730c9f6f43af0469a62136553b2833000dab36001925b2
Oct 31 16:29:28.933198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.933172    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"
Oct 31 16:29:28.933348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.933191    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" event={"ID":"8ce2dce3-4cfd-476b-b812-1c0926b8b88a","Type":"ContainerDied","Data":"1dc806f339da90f75d8601fb32c91e106f826c1905b628caca23b22f6a4990a1"}
Oct 31 16:29:28.933348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.933263    4417 scope.go:122] "RemoveContainer" containerID="1ce30bdf30b54ff325acdc7c06ff214378f7802f10db68cc4b890598e324757f"
Oct 31 16:29:28.935103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.935077    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" event={"ID":"515acf0c-2a58-44f1-92bc-ae872d52dd3a","Type":"ContainerStarted","Data":"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"}
Oct 31 16:29:28.935229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.935128    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" event={"ID":"515acf0c-2a58-44f1-92bc-ae872d52dd3a","Type":"ContainerStarted","Data":"2b60e76e0597d2e6339d66df1e7e2fe0fec9d7d07904a1a824d28b4b442ea300"}
Oct 31 16:29:28.935229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.935156    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:28.936833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.936809    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"
Oct 31 16:29:28.936940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.936858    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl" event={"ID":"90b0f869-a6fa-4bb5-82fc-eccf8a08f759","Type":"ContainerDied","Data":"e1a1dcc733542ac49891010f0d853d59993d58b1c313a277ae20d2af91c3b113"}
Oct 31 16:29:28.945010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.944824    4417 generic.go:356] "Generic (PLEG): container finished" podID="80599a52-229a-4d25-a347-687580a623e8" containerID="2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627" exitCode=0
Oct 31 16:29:28.945010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.944882    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"
Oct 31 16:29:28.945010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.944924    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerDied","Data":"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"}
Oct 31 16:29:28.945010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.944971    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r" event={"ID":"80599a52-229a-4d25-a347-687580a623e8","Type":"ContainerDied","Data":"9041f07d80b637cc23df2be88dcf59ed9668f11940b28e553e5c84dca343cbd0"}
Oct 31 16:29:28.949113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.948368    4417 generic.go:356] "Generic (PLEG): container finished" podID="393b607a-4354-403f-a966-fb91dc414bf7" containerID="1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3" exitCode=0
Oct 31 16:29:28.949113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.948481    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" event={"ID":"393b607a-4354-403f-a966-fb91dc414bf7","Type":"ContainerDied","Data":"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"}
Oct 31 16:29:28.949113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.948523    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq" event={"ID":"393b607a-4354-403f-a966-fb91dc414bf7","Type":"ContainerDied","Data":"47410adf07ced1fadfe757ce3a271335f75b4116ff72bbff8da8839455b561c3"}
Oct 31 16:29:28.949113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.948596    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"
Oct 31 16:29:28.953429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.953354    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" event={"ID":"6de4c19b-ad77-407d-be89-6ac94284fe85","Type":"ContainerStarted","Data":"1ddd0b05f2a2fef483730c9f6f43af0469a62136553b2833000dab36001925b2"}
Oct 31 16:29:28.953514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.953447    4417 scope.go:122] "RemoveContainer" containerID="47d5d7ad276c262fadd8bea255a0bb581212381cebf279903b41fedc00d229ae"
Oct 31 16:29:28.976665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.963763    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv" event={"ID":"1baba935-5cd0-44bd-a815-f5742357b8b6","Type":"ContainerDied","Data":"df685fe86f2587e5995598a0579e48eca5e5d1e0f92dd5a3b566071124d2bc47"}
Oct 31 16:29:28.976665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.963886    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"
Oct 31 16:29:28.976665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.968313    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf" event={"ID":"408aa05b-4a02-4177-ad74-ffa930bd8dc2","Type":"ContainerDied","Data":"cb4f8657660831c308874a1a22e31ae8833a665ae6a61810bb1690ce8a36b7a3"}
Oct 31 16:29:28.976665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.968418    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"
Oct 31 16:29:28.987413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.981484    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj" event={"ID":"def7b5ed-7f7a-48d2-b253-a5e06e0272ca","Type":"ContainerDied","Data":"e3d1467d24118f68777ffe4e23d92ff7cefa008acbda4514ae995255dff9a3a7"}
Oct 31 16:29:28.987413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.981598    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"
Oct 31 16:29:28.993927 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.993903    4417 scope.go:122] "RemoveContainer" containerID="2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"
Oct 31 16:29:28.994218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.994188    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv" event={"ID":"72cdad85-dd92-4e37-9562-2acd458fc1d8","Type":"ContainerDied","Data":"362de64bda35878486c4a825df83b37456dcd9762600279d7545d85aa267b307"}
Oct 31 16:29:28.994306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:28.994292    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"
Oct 31 16:29:29.004285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.004260    4417 generic.go:356] "Generic (PLEG): container finished" podID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerID="574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad" exitCode=0
Oct 31 16:29:29.004399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.004383    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"
Oct 31 16:29:29.004463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.004412    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" event={"ID":"dead36ef-9851-4d9a-a53a-9df732c6303a","Type":"ContainerDied","Data":"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"}
Oct 31 16:29:29.004526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.004468    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd" event={"ID":"dead36ef-9851-4d9a-a53a-9df732c6303a","Type":"ContainerDied","Data":"a7d41888c97d32420e934cfe84e7aa12514ba7588dce82509a34fc0d037e11e5"}
Oct 31 16:29:29.006588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.006568    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9" event={"ID":"58203f05-1714-450a-a50c-42592d06eb5f","Type":"ContainerDied","Data":"22259e96f40b8ff94b1b50c0905580619ee52f07dbe6387fea3df3a2034ebe0a"}
Oct 31 16:29:29.007171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.007152    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"
Oct 31 16:29:29.013340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.013266    4417 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:29:29.013502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.013482    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:29:29.014875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.014848    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" event={"ID":"8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74","Type":"ContainerDied","Data":"a72874b69c4b77db188bd8d4b530b174ad60db1a2d56412503b2875aaf8ece38"}
Oct 31 16:29:29.014983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.014951    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"
Oct 31 16:29:29.019763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.019740    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x" event={"ID":"7161f702-5ab0-4b78-9f49-00d10a01557d","Type":"ContainerDied","Data":"b13ae23efd08d4db3a9ef61f0bf93c1c0577b7a8f0895d5cd490d042e6ad121d"}
Oct 31 16:29:29.019883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.019772    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"
Oct 31 16:29:29.135294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.135272    4417 scope.go:122] "RemoveContainer" containerID="62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"
Oct 31 16:29:29.154325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.154300    4417 scope.go:122] "RemoveContainer" containerID="2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"
Oct 31 16:29:29.155940 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:29.155682    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627\": container with ID starting with 2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627 not found: ID does not exist" containerID="2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"
Oct 31 16:29:29.155940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.155713    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627"} err="failed to get container status \"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627\": rpc error: code = NotFound desc = could not find container \"2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627\": container with ID starting with 2d71fe56bf98596a7f0c6356821015b5b553d21fdf6a107494d1ebea0c5e7627 not found: ID does not exist"
Oct 31 16:29:29.155940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.155732    4417 scope.go:122] "RemoveContainer" containerID="62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"
Oct 31 16:29:29.156157 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:29.155989    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a\": container with ID starting with 62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a not found: ID does not exist" containerID="62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"
Oct 31 16:29:29.156157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.156018    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a"} err="failed to get container status \"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a\": rpc error: code = NotFound desc = could not find container \"62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a\": container with ID starting with 62c9bc0657bd7b7b0b81251853644d73bfed0fdfa2c0c48fda252ce3e439954a not found: ID does not exist"
Oct 31 16:29:29.156157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.156041    4417 scope.go:122] "RemoveContainer" containerID="1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"
Oct 31 16:29:29.170536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.170506    4417 scope.go:122] "RemoveContainer" containerID="1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"
Oct 31 16:29:29.170946 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:29.170914    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3\": container with ID starting with 1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3 not found: ID does not exist" containerID="1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"
Oct 31 16:29:29.171051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.170960    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3"} err="failed to get container status \"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3\": rpc error: code = NotFound desc = could not find container \"1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3\": container with ID starting with 1fbd6fe9e4b1bbb277f3b008093fdf8ee7c9095b39c0dbeca8e2886afb8b1ef3 not found: ID does not exist"
Oct 31 16:29:29.171051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.170984    4417 scope.go:122] "RemoveContainer" containerID="001eb2c700e867dac46f77d9ff90902fc47b9b27eeddb1751f195254e17510ae"
Oct 31 16:29:29.188863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.181013    4417 scope.go:122] "RemoveContainer" containerID="a762102448a8a0e16610cad924b0e6202a3b8acef27c234df841e760cff3db0a"
Oct 31 16:29:29.311926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.308521    4417 scope.go:122] "RemoveContainer" containerID="12b0f14580a0bcf2d2b541c6dc3b1aea4fcc7f6674d8e5dc1c2271c0fdaf9e92"
Oct 31 16:29:29.335878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.335840    4417 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:29:29.336117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.336093    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:29:29.465093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.465060    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:29:29.477484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.477449    4417 cpu_manager.go:414] "containerMap: removing container" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623819    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623884    4417 cpu_manager.go:414] "containerMap: removing container" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623893    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623903    4417 cpu_manager.go:414] "containerMap: removing container" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623911    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623931    4417 cpu_manager.go:414] "containerMap: removing container" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623938    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623965    4417 cpu_manager.go:414] "containerMap: removing container" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.623972    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624017    4417 cpu_manager.go:414] "containerMap: removing container" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624025    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624036    4417 cpu_manager.go:414] "containerMap: removing container" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624045    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624063    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624069    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624090    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624096    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624105    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624111    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624127    4417 cpu_manager.go:414] "containerMap: removing container" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624133    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624143    4417 cpu_manager.go:414] "containerMap: removing container" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624149    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624161    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624168    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624193    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624200    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624215    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624222    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624470    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624491    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624503    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624514    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624524    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624535    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624544    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624553    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624562    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="393b607a-4354-403f-a966-fb91dc414bf7" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624573    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="80599a52-229a-4d25-a347-687580a623e8" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624590    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="58203f05-1714-450a-a50c-42592d06eb5f" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624597    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624609    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" containerName="istio-proxy"
Oct 31 16:29:29.624698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.624636    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" containerName="istio-proxy"
Oct 31 16:29:29.639655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.492346    4417 scope.go:122] "RemoveContainer" containerID="0f88b900f3de183f042cfee56b489317b3f4506f4935bbeba4c63eea840697a2"
Oct 31 16:29:29.651567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.651535    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:29:29.786028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.785979    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:29:29.786926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.676823    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.790449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.788438    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" containerName="istio-proxy"
Oct 31 16:29:29.825583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.825326    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:29:29.825583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.825461    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:29.846675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.842166    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dac4fa\""
Oct 31 16:29:29.846675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.842306    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dac4fa-dockercfg-s8x5r\""
Oct 31 16:29:29.846675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.844796    4417 scope.go:122] "RemoveContainer" containerID="574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"
Oct 31 16:29:29.877305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.877280    4417 scope.go:122] "RemoveContainer" containerID="574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"
Oct 31 16:29:29.877911 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:29.877869    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad\": container with ID starting with 574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad not found: ID does not exist" containerID="574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"
Oct 31 16:29:29.878017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.877922    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad"} err="failed to get container status \"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad\": rpc error: code = NotFound desc = could not find container \"574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad\": container with ID starting with 574506b63032cc04e369bb8c1d160fc52f82f4bf3aba300cbf629b717ce4d7ad not found: ID does not exist"
Oct 31 16:29:29.878017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.877947    4417 scope.go:122] "RemoveContainer" containerID="50d76eafa7a1a762b80e10038acd1cf10ff44c51b5d7d3d024d5f0ed1747c710"
Oct 31 16:29:29.882177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.882149    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mz7\" (UniqueName: \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.882566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.882539    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.888542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.888521    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:29:29.893525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.893501    4417 scope.go:122] "RemoveContainer" containerID="01c189ae069174233ba0551bde129eeaa23c8febd5273c453bc757a4b7bce79a"
Oct 31 16:29:29.897229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.897206    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-ldg9x"]
Oct 31 16:29:29.904251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.904234    4417 scope.go:122] "RemoveContainer" containerID="ae6b7a60d2391780d3f8738654ae7bd194e362c46019cd634000465a2f18f138"
Oct 31 16:29:29.984296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.984261    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrcs\" (UniqueName: \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:29.984475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.984314    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mz7\" (UniqueName: \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.984574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.984551    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:29.984722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.984699    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.985415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.985396    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:29.996126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:29.996083    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:29:30.000517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.000489    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-v2nnf"]
Oct 31 16:29:30.028278 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.028239    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" event={"ID":"6de4c19b-ad77-407d-be89-6ac94284fe85","Type":"ContainerStarted","Data":"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"}
Oct 31 16:29:30.028452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.028332    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:30.037110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.037083    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mz7\" (UniqueName: \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\") pod \"payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:30.086632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.086585    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:30.086872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.086749    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrcs\" (UniqueName: \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:30.087303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.087284    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:30.098856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.098824    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:29:30.102922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.102893    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fqnvv"]
Oct 31 16:29:30.140552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.140472    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrcs\" (UniqueName: \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\") pod \"payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") " pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:30.166031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.166002    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:30.174884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.174859    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:30.206843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.206563    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:29:30.214286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.214252    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-22w7r"]
Oct 31 16:29:30.316280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.316249    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:29:30.320794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.320736    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-95wnl"]
Oct 31 16:29:30.420335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.420227    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:29:30.424404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.424374    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-8b7jd"]
Oct 31 16:29:30.523358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.523298    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" podStartSLOduration=4.523285225 podStartE2EDuration="4.523285225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:30.522655546 +0000 UTC m=+1615.584230723" watchObservedRunningTime="2026-10-31 16:29:30.523285225 +0000 UTC m=+1615.584860410"
Oct 31 16:29:30.545956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.545922    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:29:30.547952 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:30.547921    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e7c86e_1e1d_4261_b953_193b3c066923.slice/crio-c3d64ea5baaadf275fa1a8b059008fc29dec657a436a013ee33d0f5ef1f9c2d1 WatchSource:0}: Error finding container c3d64ea5baaadf275fa1a8b059008fc29dec657a436a013ee33d0f5ef1f9c2d1: Status 404 returned error can't find the container with id c3d64ea5baaadf275fa1a8b059008fc29dec657a436a013ee33d0f5ef1f9c2d1
Oct 31 16:29:30.558284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.558256    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:29:30.576072 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:30.576043    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e77561_0a8c_4e83_9524_187cef29f604.slice/crio-e4626808523da5603b3dd830b317ee18a43cd8aa01d7cda43bc8d0bf4a2e7245 WatchSource:0}: Error finding container e4626808523da5603b3dd830b317ee18a43cd8aa01d7cda43bc8d0bf4a2e7245: Status 404 returned error can't find the container with id e4626808523da5603b3dd830b317ee18a43cd8aa01d7cda43bc8d0bf4a2e7245
Oct 31 16:29:30.627968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.627940    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:29:30.631676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.631645    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-hc5lj"]
Oct 31 16:29:30.736508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.736438    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:29:30.781419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.780714    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-5b2hv"]
Oct 31 16:29:30.837593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.837557    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:29:30.843788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.843748    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-hgnhc"]
Oct 31 16:29:30.943987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.943951    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:29:30.949366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:30.949331    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-b9ls9"]
Oct 31 16:29:31.059994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.059965    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1baba935-5cd0-44bd-a815-f5742357b8b6" path="/var/lib/kubelet/pods/1baba935-5cd0-44bd-a815-f5742357b8b6/volumes"
Oct 31 16:29:31.060662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.060638    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408aa05b-4a02-4177-ad74-ffa930bd8dc2" path="/var/lib/kubelet/pods/408aa05b-4a02-4177-ad74-ffa930bd8dc2/volumes"
Oct 31 16:29:31.061171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.061156    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58203f05-1714-450a-a50c-42592d06eb5f" path="/var/lib/kubelet/pods/58203f05-1714-450a-a50c-42592d06eb5f/volumes"
Oct 31 16:29:31.061941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.061926    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7161f702-5ab0-4b78-9f49-00d10a01557d" path="/var/lib/kubelet/pods/7161f702-5ab0-4b78-9f49-00d10a01557d/volumes"
Oct 31 16:29:31.062360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.062348    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cdad85-dd92-4e37-9562-2acd458fc1d8" path="/var/lib/kubelet/pods/72cdad85-dd92-4e37-9562-2acd458fc1d8/volumes"
Oct 31 16:29:31.062809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.062795    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80599a52-229a-4d25-a347-687580a623e8" path="/var/lib/kubelet/pods/80599a52-229a-4d25-a347-687580a623e8/volumes"
Oct 31 16:29:31.063540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.063526    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74" path="/var/lib/kubelet/pods/8ebc57bf-e8b0-46e9-9c71-09cb0f8fcf74/volumes"
Oct 31 16:29:31.063974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.063960    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b0f869-a6fa-4bb5-82fc-eccf8a08f759" path="/var/lib/kubelet/pods/90b0f869-a6fa-4bb5-82fc-eccf8a08f759/volumes"
Oct 31 16:29:31.064659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.064644    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dead36ef-9851-4d9a-a53a-9df732c6303a" path="/var/lib/kubelet/pods/dead36ef-9851-4d9a-a53a-9df732c6303a/volumes"
Oct 31 16:29:31.065059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065046    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def7b5ed-7f7a-48d2-b253-a5e06e0272ca" path="/var/lib/kubelet/pods/def7b5ed-7f7a-48d2-b253-a5e06e0272ca/volumes"
Oct 31 16:29:31.065355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065338    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" event={"ID":"56e77561-0a8c-4e83-9524-187cef29f604","Type":"ContainerStarted","Data":"4559efb64a26ea52355cd4e5336b300869433392a6f5cc7395a3c0179d8c4d40"}
Oct 31 16:29:31.065396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065383    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" event={"ID":"56e77561-0a8c-4e83-9524-187cef29f604","Type":"ContainerStarted","Data":"e4626808523da5603b3dd830b317ee18a43cd8aa01d7cda43bc8d0bf4a2e7245"}
Oct 31 16:29:31.065428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065409    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:29:31.065461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065429    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-68tlq"]
Oct 31 16:29:31.065461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065451    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" event={"ID":"c0e7c86e-1e1d-4261-b953-193b3c066923","Type":"ContainerStarted","Data":"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"}
Oct 31 16:29:31.065523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065479    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" event={"ID":"c0e7c86e-1e1d-4261-b953-193b3c066923","Type":"ContainerStarted","Data":"c3d64ea5baaadf275fa1a8b059008fc29dec657a436a013ee33d0f5ef1f9c2d1"}
Oct 31 16:29:31.065523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065500    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:31.065523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.065516    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:31.148244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.148202    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:29:31.153626 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.153593    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-l9fmz"]
Oct 31 16:29:31.257844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.257737    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" podStartSLOduration=5.257719369 podStartE2EDuration="5.257719369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:31.253604135 +0000 UTC m=+1616.315179319" watchObservedRunningTime="2026-10-31 16:29:31.257719369 +0000 UTC m=+1616.319294553"
Oct 31 16:29:31.358325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.358198    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" podStartSLOduration=2.358179411 podStartE2EDuration="2.358179411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:31.356377047 +0000 UTC m=+1616.417952231" watchObservedRunningTime="2026-10-31 16:29:31.358179411 +0000 UTC m=+1616.419754598"
Oct 31 16:29:31.464639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.464565    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" podStartSLOduration=2.464550587 podStartE2EDuration="2.464550587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:31.461467959 +0000 UTC m=+1616.523043144" watchObservedRunningTime="2026-10-31 16:29:31.464550587 +0000 UTC m=+1616.526125772"
Oct 31 16:29:31.772693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.770777    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:29:31.801463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.801423    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:31.812653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.810135    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhn7\" (UniqueName: \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:31.812653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.810286    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:31.841669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.838596    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:29:31.861678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.861482    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-f43174\""
Oct 31 16:29:31.861678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.861645    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-f43174-dockercfg-glxqt\""
Oct 31 16:29:32.010533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.977106    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:32.015026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.014959    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhn7\" (UniqueName: \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:32.035075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:31.987492    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:32.137312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.137258    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:29:32.199348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.199315    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.203806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.199211    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:29:32.205518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.205490    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhn7\" (UniqueName: \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\") pod \"payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:32.363097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.363054    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.363374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.363349    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prm2\" (UniqueName: \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.432501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.432461    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:32.465074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.465023    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.465252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.465090    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prm2\" (UniqueName: \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.466213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.466183    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.518656 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.518571    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prm2\" (UniqueName: \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\") pod \"payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.533280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.533252    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:32.814942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.814911    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:29:32.816436 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:32.816409    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb5c016_a6dc_49fb_8974_d4c2fecba42f.slice/crio-35b4b921953a60d2d17f75e06cb8f81a2aba24edbeeb62da3e7ba6b68553b257 WatchSource:0}: Error finding container 35b4b921953a60d2d17f75e06cb8f81a2aba24edbeeb62da3e7ba6b68553b257: Status 404 returned error can't find the container with id 35b4b921953a60d2d17f75e06cb8f81a2aba24edbeeb62da3e7ba6b68553b257
Oct 31 16:29:32.919145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:32.919119    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:29:32.920658 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:32.920598    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb623fcaa_64b7_41ab_93e7_a98727c4235e.slice/crio-e0c3a92d5612225409bb56392101ed2301cd0b15acdf70296aa09a6d50882ffc WatchSource:0}: Error finding container e0c3a92d5612225409bb56392101ed2301cd0b15acdf70296aa09a6d50882ffc: Status 404 returned error can't find the container with id e0c3a92d5612225409bb56392101ed2301cd0b15acdf70296aa09a6d50882ffc
Oct 31 16:29:32.961847 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:32.961817    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:29:33.051954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.051919    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393b607a-4354-403f-a966-fb91dc414bf7" path="/var/lib/kubelet/pods/393b607a-4354-403f-a966-fb91dc414bf7/volumes"
Oct 31 16:29:33.052692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.052674    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce2dce3-4cfd-476b-b812-1c0926b8b88a" path="/var/lib/kubelet/pods/8ce2dce3-4cfd-476b-b812-1c0926b8b88a/volumes"
Oct 31 16:29:33.213540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.213407    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" event={"ID":"b623fcaa-64b7-41ab-93e7-a98727c4235e","Type":"ContainerStarted","Data":"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"}
Oct 31 16:29:33.213540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.213485    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:33.213540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.213510    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" event={"ID":"b623fcaa-64b7-41ab-93e7-a98727c4235e","Type":"ContainerStarted","Data":"e0c3a92d5612225409bb56392101ed2301cd0b15acdf70296aa09a6d50882ffc"}
Oct 31 16:29:33.216034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.216006    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" event={"ID":"feb5c016-a6dc-49fb-8974-d4c2fecba42f","Type":"ContainerStarted","Data":"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"}
Oct 31 16:29:33.216180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.216052    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" event={"ID":"feb5c016-a6dc-49fb-8974-d4c2fecba42f","Type":"ContainerStarted","Data":"35b4b921953a60d2d17f75e06cb8f81a2aba24edbeeb62da3e7ba6b68553b257"}
Oct 31 16:29:33.293966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.293934    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:29:33.312316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.312285    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:29:33.320896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.320849    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" podStartSLOduration=2.320836789 podStartE2EDuration="2.320836789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:33.31838749 +0000 UTC m=+1618.379962680" watchObservedRunningTime="2026-10-31 16:29:33.320836789 +0000 UTC m=+1618.382412024"
Oct 31 16:29:33.571687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:33.560963    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" podStartSLOduration=2.560935536 podStartE2EDuration="2.560935536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:33.534091458 +0000 UTC m=+1618.595666645" watchObservedRunningTime="2026-10-31 16:29:33.560935536 +0000 UTC m=+1618.622510722"
Oct 31 16:29:34.249913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:34.249877    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:39.168031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.167989    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:29:39.174793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.174768    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.222756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.222698    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqn89\" (UniqueName: \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.222940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.222763    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.223010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.222935    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.223067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.223005    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.223296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.223278    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a1b30a15-serving-cert\""
Oct 31 16:29:39.274298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.274256    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:29:39.326959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.326921    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn89\" (UniqueName: \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.327156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.326965    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.327156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.327070    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.327156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.327109    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.327450 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:39.327416    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-serving-cert: secret "maas-api-e2e-mt-a1b30a15-serving-cert" not found
Oct 31 16:29:39.327682 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:39.327485    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls podName:11daf9e6-002c-444d-88d8-1f666b55ce84 nodeName:}" failed. No retries permitted until 2026-10-31 16:29:39.827468799 +0000 UTC m=+1624.889043970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls") pod "maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" (UID: "11daf9e6-002c-444d-88d8-1f666b55ce84") : secret "maas-api-e2e-mt-a1b30a15-serving-cert" not found
Oct 31 16:29:39.328338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.328315    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.335077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.335050    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.384346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.384312    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqn89\" (UniqueName: \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.867720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.867671    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:39.873306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:39.873272    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:40.043902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:40.043869    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:29:40.089137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:40.089102    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:40.506690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:40.506607    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:29:40.507263 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:40.507240    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11daf9e6_002c_444d_88d8_1f666b55ce84.slice/crio-84c230189ef32a08b296541172c2a963c832d4464df83e2c1e36361df9a6594f WatchSource:0}: Error finding container 84c230189ef32a08b296541172c2a963c832d4464df83e2c1e36361df9a6594f: Status 404 returned error can't find the container with id 84c230189ef32a08b296541172c2a963c832d4464df83e2c1e36361df9a6594f
Oct 31 16:29:40.508865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:40.508847    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:29:41.064144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:41.064115    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:29:41.445205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:41.437244    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" event={"ID":"11daf9e6-002c-444d-88d8-1f666b55ce84","Type":"ContainerStarted","Data":"84c230189ef32a08b296541172c2a963c832d4464df83e2c1e36361df9a6594f"}
Oct 31 16:29:42.206546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:42.206516    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:29:42.501505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:42.501416    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" event={"ID":"11daf9e6-002c-444d-88d8-1f666b55ce84","Type":"ContainerStarted","Data":"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"}
Oct 31 16:29:42.502119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:42.501846    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:42.609849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:42.609775    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" podStartSLOduration=2.687002335 podStartE2EDuration="3.609758553s" totalImagesPullingTime="922.756218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:42.607038962 +0000 UTC m=+1627.668614144" watchObservedRunningTime="2026-10-31 16:29:42.609758553 +0000 UTC m=+1627.671333762"
Oct 31 16:29:43.219416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:43.219384    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:29:44.258599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:44.258567    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:29:45.259627 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:45.259593    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:29:45.922495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:45.922462    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:29:48.543451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:48.543417    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:29:50.923256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:50.923216    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" podUID="6b385f77-5319-469c-acd9-a4336c384293" containerName="payload-processing" containerID="cri-o://7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96" gracePeriod=25
Oct 31 16:29:51.195202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.195140    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:29:51.317036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.316998    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\") pod \"6b385f77-5319-469c-acd9-a4336c384293\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") "
Oct 31 16:29:51.317239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.317134    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\" (UniqueName: \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\") pod \"6b385f77-5319-469c-acd9-a4336c384293\" (UID: \"6b385f77-5319-469c-acd9-a4336c384293\") "
Oct 31 16:29:51.317484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.317460    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume" pod "6b385f77-5319-469c-acd9-a4336c384293" (UID: "6b385f77-5319-469c-acd9-a4336c384293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:51.317894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.317860    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b385f77-5319-469c-acd9-a4336c384293-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:51.329155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.329124    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj" pod "6b385f77-5319-469c-acd9-a4336c384293" (UID: "6b385f77-5319-469c-acd9-a4336c384293"). InnerVolumeSpecName "kube-api-access-qxvzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:51.426092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.425990    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvzj\" (UniqueName: \"kubernetes.io/projected/6b385f77-5319-469c-acd9-a4336c384293-kube-api-access-qxvzj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:29:51.710675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.710598    4417 generic.go:356] "Generic (PLEG): container finished" podID="6b385f77-5319-469c-acd9-a4336c384293" containerID="7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96" exitCode=0
Oct 31 16:29:51.710837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.710691    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"
Oct 31 16:29:51.710837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.710695    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" event={"ID":"6b385f77-5319-469c-acd9-a4336c384293","Type":"ContainerDied","Data":"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"}
Oct 31 16:29:51.710837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.710771    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697" event={"ID":"6b385f77-5319-469c-acd9-a4336c384293","Type":"ContainerDied","Data":"225db0ba3f88b49ef8e400b087c1aceb815cb1ab8e13f0249baf8b5eb113a699"}
Oct 31 16:29:51.710837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.710794    4417 scope.go:122] "RemoveContainer" containerID="7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"
Oct 31 16:29:51.721456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.721437    4417 scope.go:122] "RemoveContainer" containerID="7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"
Oct 31 16:29:51.721808 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:29:51.721768    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96\": container with ID starting with 7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96 not found: ID does not exist" containerID="7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"
Oct 31 16:29:51.721882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.721817    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96"} err="failed to get container status \"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96\": rpc error: code = NotFound desc = could not find container \"7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96\": container with ID starting with 7af5f6b0069a58c07766552eeb10ac69b1796f3b73e7161e4efeced95b731e96 not found: ID does not exist"
Oct 31 16:29:51.863729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.863678    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:29:51.868020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:51.867987    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a7907d70-f46b98989-sb697"]
Oct 31 16:29:52.326855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.326822    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:29:52.327792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.327772    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6b385f77-5319-469c-acd9-a4336c384293" containerName="payload-processing"
Oct 31 16:29:52.327869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.327794    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b385f77-5319-469c-acd9-a4336c384293" containerName="payload-processing"
Oct 31 16:29:52.327969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.327949    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b385f77-5319-469c-acd9-a4336c384293" containerName="payload-processing"
Oct 31 16:29:52.332772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.332750    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.385856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.383750    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-705a67-openshift-default-dockercfg-mn68c\""
Oct 31 16:29:52.422671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.413751    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454463    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454609    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454672    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454701    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9z5w\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454750    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454784    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.454990    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.455076    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.462641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.455109    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.598031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.597286    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.598031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.597349    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.598031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.597463    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.598031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.597499    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.598031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.597533    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.602562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.601706    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.602562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.602496    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.603155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.603119    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.603380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.603354    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.604026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.604003    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.604846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.604760    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.605128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.605109    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.606013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.605919    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.606832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.606808    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9z5w\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.635640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.628392    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.649735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.648583    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.675224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.671088    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.726676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.700489    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9z5w\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\") pod \"e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") " pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:52.958183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:52.958093    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:53.053505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.053475    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b385f77-5319-469c-acd9-a4336c384293" path="/var/lib/kubelet/pods/6b385f77-5319-469c-acd9-a4336c384293/volumes"
Oct 31 16:29:53.381536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.381504    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:29:53.383727 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:29:53.383694    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc3cb4c_9227_4c52_a024_b9564d7d06e4.slice/crio-5a4de262da210b6e4f6affdad16a9fabae1a2a5701e65910fc6909b5ace2029b WatchSource:0}: Error finding container 5a4de262da210b6e4f6affdad16a9fabae1a2a5701e65910fc6909b5ace2029b: Status 404 returned error can't find the container with id 5a4de262da210b6e4f6affdad16a9fabae1a2a5701e65910fc6909b5ace2029b
Oct 31 16:29:53.386812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.386780    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:53.386905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.386849    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:53.386905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.386878    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:29:53.765807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.765696    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerStarted","Data":"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"}
Oct 31 16:29:53.765807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.765765    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerStarted","Data":"5a4de262da210b6e4f6affdad16a9fabae1a2a5701e65910fc6909b5ace2029b"}
Oct 31 16:29:53.919266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.919207    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podStartSLOduration=1.919192324 podStartE2EDuration="1.919192324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:29:53.918360235 +0000 UTC m=+1638.979935420" watchObservedRunningTime="2026-10-31 16:29:53.919192324 +0000 UTC m=+1638.980767509"
Oct 31 16:29:53.959203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:53.959165    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:54.965494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:54.965461    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:55.780244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:55.780214    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:29:55.781343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:29:55.781322    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:30:00.218642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.215540    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:00.229670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.227293    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.248739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.245123    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:00.256958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.256924    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.270647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.265813    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8lb\" (UniqueName: \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.270647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.265884    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.294190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.294154    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:00.336020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.335740    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:00.338460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.337896    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.345490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.345461    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:00.345930 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.345830    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.355209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.355170    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:30:00.355344 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.355328    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:00.355421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.355404    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:00.355478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.355434    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:00.355531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.355522    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.360249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.360226    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:00.368383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.367670    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.368383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.367711    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhzg\" (UniqueName: \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.368383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.367887    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8lb\" (UniqueName: \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.368383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.367950    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.368383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.368318    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwqf\" (UniqueName: \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.369082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.368684    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.369082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.368828    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqt5\" (UniqueName: \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.369082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.368904    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.369403 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.369378    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.416334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.415873    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:00.425074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.423450    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8lb\" (UniqueName: \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\") pod \"maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.470160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470060    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqt5\" (UniqueName: \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.470160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470138    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.470386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470243    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.470428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470398    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.470468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470446    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhzg\" (UniqueName: \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.470520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470504    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ht42\" (UniqueName: \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.470602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470585    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.470713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.470684    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwqf\" (UniqueName: \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.471047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.471017    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.471130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.471114    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.471276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.471257    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.523317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.523278    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwqf\" (UniqueName: \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.523598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.523576    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqt5\" (UniqueName: \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.523738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.523721    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhzg\" (UniqueName: \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.590691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.590330    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:00.593314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.593275    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.593523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.593419    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ht42\" (UniqueName: \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.595209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.595172    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:00.650716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.650242    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:00.732690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.706536    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:00.732690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.708816    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:00.732690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.731975    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ht42\" (UniqueName: \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\") pod \"maas-api-key-cleanup-29891070-9nxk5\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:01.018928 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:00.979550    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:30:01.107265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.086113    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.107265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.103336    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:01.151638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.145855    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-705a67\""
Oct 31 16:30:01.151638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.146609    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-705a67-dockercfg-4bzvt\""
Oct 31 16:30:01.465701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.323857    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2mp\" (UniqueName: \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.465701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.324179    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.465701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.430585    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2mp\" (UniqueName: \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.465701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.460478    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.465701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.461586    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.489909 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.489873    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:30:01.490087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.489927    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:30:01.500841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.500798    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:30:01.501049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.500928    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.517331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.517286    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:01.536440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.535806    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2mp\" (UniqueName: \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\") pod \"payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.547805 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:01.547753    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db21af1_60b6_47fc_9647_c8740b6a90b0.slice/crio-a577f0dad483258290d3b69ab20cac94741391d3744b118003fe277be75b8ee1 WatchSource:0}: Error finding container a577f0dad483258290d3b69ab20cac94741391d3744b118003fe277be75b8ee1: Status 404 returned error can't find the container with id a577f0dad483258290d3b69ab20cac94741391d3744b118003fe277be75b8ee1
Oct 31 16:30:01.565003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.562095    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5mn\" (UniqueName: \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.565003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.562163    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.663811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.663779    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5mn\" (UniqueName: \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.664013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.663832    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.664587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.664567    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.716454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.716423    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5mn\" (UniqueName: \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\") pod \"payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.789883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.789840    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:01.820954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.820923    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:01.877442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.873053    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:01.880263 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:01.880232    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7528865b_c72b_4b18_b770_853e4284d4c9.slice/crio-5ba2eebe160aca4db353f9fce7c4fd6721fe09328d2a7f688bfdebe95b516ad5 WatchSource:0}: Error finding container 5ba2eebe160aca4db353f9fce7c4fd6721fe09328d2a7f688bfdebe95b516ad5: Status 404 returned error can't find the container with id 5ba2eebe160aca4db353f9fce7c4fd6721fe09328d2a7f688bfdebe95b516ad5
Oct 31 16:30:01.880465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.880449    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:01.880893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.880874    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:01.882888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:01.882866    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:02.109843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.109719    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerStarted","Data":"1c4e226cc30a8e285ba93bd38b1e253e856e2d6e28eca559a763656748aecc1b"}
Oct 31 16:30:02.111188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.111156    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerStarted","Data":"c3cb2ee2ed06bb57c989a406acb4261c6a0ee7116af87531c52da9567e71282b"}
Oct 31 16:30:02.112497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.112476    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerStarted","Data":"5ba2eebe160aca4db353f9fce7c4fd6721fe09328d2a7f688bfdebe95b516ad5"}
Oct 31 16:30:02.113661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.113609    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerStarted","Data":"1c9cba3921e597fc5e399c38bef1466278667f5cce781cc5370a42be0883ae85"}
Oct 31 16:30:02.115183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.115160    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerStarted","Data":"45358e126e7f1341f2603cba07235f85954a8a33b6786b6f67fb1453eeb6d819"}
Oct 31 16:30:02.115274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.115201    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerStarted","Data":"a577f0dad483258290d3b69ab20cac94741391d3744b118003fe277be75b8ee1"}
Oct 31 16:30:02.172502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.172374    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:30:02.175792 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:02.175762    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0470c18a_b968_40e7_bc2d_8092243c6719.slice/crio-99561aa2065865e6a419e6823583e072bf3e37fa5b91560dbc945d8534a80689 WatchSource:0}: Error finding container 99561aa2065865e6a419e6823583e072bf3e37fa5b91560dbc945d8534a80689: Status 404 returned error can't find the container with id 99561aa2065865e6a419e6823583e072bf3e37fa5b91560dbc945d8534a80689
Oct 31 16:30:02.223381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.223338    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" podStartSLOduration=2.223324487 podStartE2EDuration="2.223324487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:02.221494767 +0000 UTC m=+1647.283069951" watchObservedRunningTime="2026-10-31 16:30:02.223324487 +0000 UTC m=+1647.284899714"
Oct 31 16:30:02.288513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:02.288475    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:30:02.294635 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:02.294575    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e2b5ec_e928_455e_b1e8_af474a1a52f9.slice/crio-be56c874cc9340c93c02e09a1558089efe452494d067cb6a7cf185f341ce4ca9 WatchSource:0}: Error finding container be56c874cc9340c93c02e09a1558089efe452494d067cb6a7cf185f341ce4ca9: Status 404 returned error can't find the container with id be56c874cc9340c93c02e09a1558089efe452494d067cb6a7cf185f341ce4ca9
Oct 31 16:30:02.951191 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:02.951157    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:30:03.123767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.123733    4417 generic.go:356] "Generic (PLEG): container finished" podID="7528865b-c72b-4b18-b770-853e4284d4c9" containerID="ba83abff64da17d33b6bfb0c0406f60ca5ddc9a66910e7f2b309a7f40b61e2da" exitCode=60
Oct 31 16:30:03.123914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.123810    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerDied","Data":"ba83abff64da17d33b6bfb0c0406f60ca5ddc9a66910e7f2b309a7f40b61e2da"}
Oct 31 16:30:03.124181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.124163    4417 scope.go:122] "RemoveContainer" containerID="ba83abff64da17d33b6bfb0c0406f60ca5ddc9a66910e7f2b309a7f40b61e2da"
Oct 31 16:30:03.125559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.125535    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerStarted","Data":"b90b219afc58c665ca83b7c8344b8a8876840aa0953dfd600c812598eb240577"}
Oct 31 16:30:03.131736 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.131687    4417 generic.go:356] "Generic (PLEG): container finished" podID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerID="45358e126e7f1341f2603cba07235f85954a8a33b6786b6f67fb1453eeb6d819" exitCode=60
Oct 31 16:30:03.131802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.131758    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerDied","Data":"45358e126e7f1341f2603cba07235f85954a8a33b6786b6f67fb1453eeb6d819"}
Oct 31 16:30:03.132072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.132050    4417 scope.go:122] "RemoveContainer" containerID="45358e126e7f1341f2603cba07235f85954a8a33b6786b6f67fb1453eeb6d819"
Oct 31 16:30:03.134167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.134135    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" event={"ID":"49e2b5ec-e928-455e-b1e8-af474a1a52f9","Type":"ContainerStarted","Data":"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"}
Oct 31 16:30:03.134269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.134189    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" event={"ID":"49e2b5ec-e928-455e-b1e8-af474a1a52f9","Type":"ContainerStarted","Data":"be56c874cc9340c93c02e09a1558089efe452494d067cb6a7cf185f341ce4ca9"}
Oct 31 16:30:03.135281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.135259    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:03.137770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.137656    4417 generic.go:356] "Generic (PLEG): container finished" podID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerID="61cb5526394f147e365ad0ac8da629956fbcb108272b9588a7bc328ec56b23b0" exitCode=60
Oct 31 16:30:03.137770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.137723    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerDied","Data":"61cb5526394f147e365ad0ac8da629956fbcb108272b9588a7bc328ec56b23b0"}
Oct 31 16:30:03.138122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.138101    4417 scope.go:122] "RemoveContainer" containerID="61cb5526394f147e365ad0ac8da629956fbcb108272b9588a7bc328ec56b23b0"
Oct 31 16:30:03.147268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.147239    4417 generic.go:356] "Generic (PLEG): container finished" podID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerID="648996cdec606730caf2658f1872ec3b5eb2345700db9bed9981a31739db7b92" exitCode=60
Oct 31 16:30:03.147387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.147316    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerDied","Data":"648996cdec606730caf2658f1872ec3b5eb2345700db9bed9981a31739db7b92"}
Oct 31 16:30:03.147722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.147698    4417 scope.go:122] "RemoveContainer" containerID="648996cdec606730caf2658f1872ec3b5eb2345700db9bed9981a31739db7b92"
Oct 31 16:30:03.150014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.149398    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" event={"ID":"0470c18a-b968-40e7-bc2d-8092243c6719","Type":"ContainerStarted","Data":"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"}
Oct 31 16:30:03.150014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.149458    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" event={"ID":"0470c18a-b968-40e7-bc2d-8092243c6719","Type":"ContainerStarted","Data":"99561aa2065865e6a419e6823583e072bf3e37fa5b91560dbc945d8534a80689"}
Oct 31 16:30:03.150014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.149702    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:03.335210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.335147    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" podStartSLOduration=3.335127341 podStartE2EDuration="3.335127341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:03.33165929 +0000 UTC m=+1648.393234473" watchObservedRunningTime="2026-10-31 16:30:03.335127341 +0000 UTC m=+1648.396702526"
Oct 31 16:30:03.442033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.439908    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" podStartSLOduration=3.439887942 podStartE2EDuration="3.439887942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:03.435082087 +0000 UTC m=+1648.496657275" watchObservedRunningTime="2026-10-31 16:30:03.439887942 +0000 UTC m=+1648.501463129"
Oct 31 16:30:03.764406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:03.764353    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" podStartSLOduration=3.764336036 podStartE2EDuration="3.764336036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:03.761037138 +0000 UTC m=+1648.822612323" watchObservedRunningTime="2026-10-31 16:30:03.764336036 +0000 UTC m=+1648.825911221"
Oct 31 16:30:04.160577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.160542    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerStarted","Data":"ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"}
Oct 31 16:30:04.162854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.162819    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerStarted","Data":"9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"}
Oct 31 16:30:04.164976 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.164950    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerStarted","Data":"56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"}
Oct 31 16:30:04.166591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.166560    4417 generic.go:356] "Generic (PLEG): container finished" podID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerID="b90b219afc58c665ca83b7c8344b8a8876840aa0953dfd600c812598eb240577" exitCode=60
Oct 31 16:30:04.166709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.166600    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerDied","Data":"b90b219afc58c665ca83b7c8344b8a8876840aa0953dfd600c812598eb240577"}
Oct 31 16:30:04.166917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.166903    4417 scope.go:122] "RemoveContainer" containerID="b90b219afc58c665ca83b7c8344b8a8876840aa0953dfd600c812598eb240577"
Oct 31 16:30:04.169425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.168826    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerStarted","Data":"e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"}
Oct 31 16:30:04.268031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.267972    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" podStartSLOduration=4.26795555 podStartE2EDuration="4.26795555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.26677983 +0000 UTC m=+1649.328355015" watchObservedRunningTime="2026-10-31 16:30:04.26795555 +0000 UTC m=+1649.329530736"
Oct 31 16:30:04.372635 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.372559    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" podStartSLOduration=4.372541809 podStartE2EDuration="4.372541809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.370281149 +0000 UTC m=+1649.431856335" watchObservedRunningTime="2026-10-31 16:30:04.372541809 +0000 UTC m=+1649.434116994"
Oct 31 16:30:04.579016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:04.578959    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" podStartSLOduration=4.57894374 podStartE2EDuration="4.57894374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:04.578580591 +0000 UTC m=+1649.640155778" watchObservedRunningTime="2026-10-31 16:30:04.57894374 +0000 UTC m=+1649.640518921"
Oct 31 16:30:05.179954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.179920    4417 generic.go:356] "Generic (PLEG): container finished" podID="7528865b-c72b-4b18-b770-853e4284d4c9" containerID="56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b" exitCode=60
Oct 31 16:30:05.180468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.180004    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerDied","Data":"56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"}
Oct 31 16:30:05.180468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.180067    4417 scope.go:122] "RemoveContainer" containerID="ba83abff64da17d33b6bfb0c0406f60ca5ddc9a66910e7f2b309a7f40b61e2da"
Oct 31 16:30:05.180577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.180511    4417 scope.go:122] "RemoveContainer" containerID="56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"
Oct 31 16:30:05.180971 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:05.180930    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2_odh-ai-gateway-infra(7528865b-c72b-4b18-b770-853e4284d4c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" podUID="7528865b-c72b-4b18-b770-853e4284d4c9"
Oct 31 16:30:05.182698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.182593    4417 generic.go:356] "Generic (PLEG): container finished" podID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerID="4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a" exitCode=60
Oct 31 16:30:05.182698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.182654    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerDied","Data":"4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a"}
Oct 31 16:30:05.183055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.183031    4417 scope.go:122] "RemoveContainer" containerID="4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a"
Oct 31 16:30:05.183457 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:05.183425    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2_odh-ai-gateway-infra(d37126e0-3210-4d6e-a389-dac6e36aacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff"
Oct 31 16:30:05.184857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.184838    4417 generic.go:356] "Generic (PLEG): container finished" podID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerID="e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd" exitCode=60
Oct 31 16:30:05.184951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.184928    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerDied","Data":"e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"}
Oct 31 16:30:05.185236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.185218    4417 scope.go:122] "RemoveContainer" containerID="e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"
Oct 31 16:30:05.185648 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:05.185605    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl_odh-ai-gateway-infra(4db21af1-60b6-47fc-9647-c8740b6a90b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0"
Oct 31 16:30:05.186933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.186915    4417 generic.go:356] "Generic (PLEG): container finished" podID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerID="ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea" exitCode=60
Oct 31 16:30:05.187036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.186981    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerDied","Data":"ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"}
Oct 31 16:30:05.187233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.187220    4417 scope.go:122] "RemoveContainer" containerID="ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"
Oct 31 16:30:05.187545 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:05.187524    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-9nxk5_odh-ai-gateway-infra(24e04d90-7091-440f-a935-7cad4db4ddd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8"
Oct 31 16:30:05.188866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.188848    4417 generic.go:356] "Generic (PLEG): container finished" podID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerID="9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69" exitCode=60
Oct 31 16:30:05.188947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.188914    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerDied","Data":"9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"}
Oct 31 16:30:05.189181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.189167    4417 scope.go:122] "RemoveContainer" containerID="9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"
Oct 31 16:30:05.189502 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:05.189482    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs_odh-ai-gateway-infra(96f14092-1638-4fd1-98c3-54c8ebf2e8c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8"
Oct 31 16:30:05.195252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.195227    4417 scope.go:122] "RemoveContainer" containerID="b90b219afc58c665ca83b7c8344b8a8876840aa0953dfd600c812598eb240577"
Oct 31 16:30:05.208562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.208540    4417 scope.go:122] "RemoveContainer" containerID="45358e126e7f1341f2603cba07235f85954a8a33b6786b6f67fb1453eeb6d819"
Oct 31 16:30:05.222426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.222408    4417 scope.go:122] "RemoveContainer" containerID="61cb5526394f147e365ad0ac8da629956fbcb108272b9588a7bc328ec56b23b0"
Oct 31 16:30:05.236600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:05.236569    4417 scope.go:122] "RemoveContainer" containerID="648996cdec606730caf2658f1872ec3b5eb2345700db9bed9981a31739db7b92"
Oct 31 16:30:06.198021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:06.197988    4417 scope.go:122] "RemoveContainer" containerID="56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"
Oct 31 16:30:06.198502 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:06.198400    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2_odh-ai-gateway-infra(7528865b-c72b-4b18-b770-853e4284d4c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" podUID="7528865b-c72b-4b18-b770-853e4284d4c9"
Oct 31 16:30:06.199750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:06.199723    4417 scope.go:122] "RemoveContainer" containerID="4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a"
Oct 31 16:30:06.200074 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:06.200052    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2_odh-ai-gateway-infra(d37126e0-3210-4d6e-a389-dac6e36aacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff"
Oct 31 16:30:06.201204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:06.201178    4417 scope.go:122] "RemoveContainer" containerID="e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"
Oct 31 16:30:06.201494 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:06.201475    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl_odh-ai-gateway-infra(4db21af1-60b6-47fc-9647-c8740b6a90b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0"
Oct 31 16:30:06.202735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:06.202718    4417 scope.go:122] "RemoveContainer" containerID="ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"
Oct 31 16:30:06.202984 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:06.202962    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-9nxk5_odh-ai-gateway-infra(24e04d90-7091-440f-a935-7cad4db4ddd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8"
Oct 31 16:30:06.204236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:06.204222    4417 scope.go:122] "RemoveContainer" containerID="9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"
Oct 31 16:30:06.204416 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:06.204400    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs_odh-ai-gateway-infra(96f14092-1638-4fd1-98c3-54c8ebf2e8c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8"
Oct 31 16:30:07.260992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.260949    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:30:07.261485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.261150    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy" containerID="cri-o://50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76" gracePeriod=30
Oct 31 16:30:07.306791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.306276    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:30:07.306791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.306514    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy" containerID="cri-o://ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b" gracePeriod=30
Oct 31 16:30:07.311337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.311283    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:30:07.311548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:07.311521    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" containerID="cri-o://10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600" gracePeriod=30
Oct 31 16:30:09.607090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.607049    4417 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:30:09.607498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.607105    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:30:09.666955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.666914    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:30:09.667233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.667179    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" containerID="cri-o://df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843" gracePeriod=30
Oct 31 16:30:09.710083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.710042    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:30:09.710334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.710297    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy" containerID="cri-o://9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d" gracePeriod=30
Oct 31 16:30:09.714536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.714507    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:30:09.714753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:09.714728    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy" containerID="cri-o://fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6" gracePeriod=30
Oct 31 16:30:12.679819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.679786    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:30:12.707407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.707383    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:30:12.710959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.710935    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:30:12.730131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730090    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.730265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730173    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.730265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730205    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.730265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730239    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730291    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730327    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730379    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730436    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.730569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730487    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.730569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730492    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.730569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730531    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730558    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730613    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730674    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730717    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730743    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730758    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730757    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730794    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730799    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:12.730856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730834    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730872    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730915    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730949    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.730995    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731019    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731038    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731096    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\") pod \"d4af8964-24c5-41e7-b001-b5e65fe721e3\" (UID: \"d4af8964-24c5-41e7-b001-b5e65fe721e3\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731129    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731196    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\") pod \"77980ac3-7ec2-475b-8191-a6988512ae66\" (UID: \"77980ac3-7ec2-475b-8191-a6988512ae66\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731231    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731294    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.731401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731358    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\") pod \"4ad1a287-7d7b-4979-ab94-124d1739061a\" (UID: \"4ad1a287-7d7b-4979-ab94-124d1739061a\") "
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731463    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731483    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731745    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.731907    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.732095    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.732100    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.732220    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:12.732661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.732486    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.732776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.732687    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733049    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733076    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77980ac3-7ec2-475b-8191-a6988512ae66-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733091    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733106    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733120    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733133    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733148    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733166    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad1a287-7d7b-4979-ab94-124d1739061a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733180    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733194    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733208    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4af8964-24c5-41e7-b001-b5e65fe721e3-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733222    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733236    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733249    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.733318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.733263    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.734551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.734512    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:12.734969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.734948    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.735081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.734976    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "kube-api-access-8plpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.735760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.735597    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.735760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.735676    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "kube-api-access-fx5wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.735760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.735729    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.736040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736005    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:12.736040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736011    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:12.736181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736042    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy" pod "4ad1a287-7d7b-4979-ab94-124d1739061a" (UID: "4ad1a287-7d7b-4979-ab94-124d1739061a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.736181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736103    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "kube-api-access-k2qjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.736181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736132    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy" pod "77980ac3-7ec2-475b-8191-a6988512ae66" (UID: "77980ac3-7ec2-475b-8191-a6988512ae66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:12.736982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.736959    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token" pod "d4af8964-24c5-41e7-b001-b5e65fe721e3" (UID: "d4af8964-24c5-41e7-b001-b5e65fe721e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:12.834855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834809    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.834855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834845    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834863    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qjf\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-kube-api-access-k2qjf\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834879    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8plpb\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-kube-api-access-8plpb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834893    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834906    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77980ac3-7ec2-475b-8191-a6988512ae66-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834920    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4af8964-24c5-41e7-b001-b5e65fe721e3-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834933    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77980ac3-7ec2-475b-8191-a6988512ae66-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834947    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77980ac3-7ec2-475b-8191-a6988512ae66-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834961    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834974    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad1a287-7d7b-4979-ab94-124d1739061a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:12.835097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:12.834987    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx5wl\" (UniqueName: \"kubernetes.io/projected/d4af8964-24c5-41e7-b001-b5e65fe721e3-kube-api-access-fx5wl\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:13.273161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.273127    4417 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:30:13.273323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.273186    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:30:13.295139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.295107    4417 generic.go:356] "Generic (PLEG): container finished" podID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerID="10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600" exitCode=0
Oct 31 16:30:13.295332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.295161    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"
Oct 31 16:30:13.295332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.295193    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" event={"ID":"4ad1a287-7d7b-4979-ab94-124d1739061a","Type":"ContainerDied","Data":"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"}
Oct 31 16:30:13.295332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.295244    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w" event={"ID":"4ad1a287-7d7b-4979-ab94-124d1739061a","Type":"ContainerDied","Data":"dc46e5d061e5c05d208384b50e0fa4dda73cb4fa343736047ebba32bb261e268"}
Oct 31 16:30:13.295332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.295265    4417 scope.go:122] "RemoveContainer" containerID="10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"
Oct 31 16:30:13.296784 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.296748    4417 generic.go:356] "Generic (PLEG): container finished" podID="77980ac3-7ec2-475b-8191-a6988512ae66" containerID="ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b" exitCode=0
Oct 31 16:30:13.296889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.296802    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"
Oct 31 16:30:13.296889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.296838    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" event={"ID":"77980ac3-7ec2-475b-8191-a6988512ae66","Type":"ContainerDied","Data":"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"}
Oct 31 16:30:13.296889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.296878    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r" event={"ID":"77980ac3-7ec2-475b-8191-a6988512ae66","Type":"ContainerDied","Data":"9d8535d8b70e7974de6e6ffbffc4bb314386afcdc46145ceb7b4d3fec5c77319"}
Oct 31 16:30:13.298558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.298537    4417 generic.go:356] "Generic (PLEG): container finished" podID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerID="50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76" exitCode=0
Oct 31 16:30:13.298703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.298559    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" event={"ID":"d4af8964-24c5-41e7-b001-b5e65fe721e3","Type":"ContainerDied","Data":"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"}
Oct 31 16:30:13.298703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.298586    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"
Oct 31 16:30:13.298703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.298597    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm" event={"ID":"d4af8964-24c5-41e7-b001-b5e65fe721e3","Type":"ContainerDied","Data":"78587a1a6622ae541f82e15d8370c55637d8242ab6564fae5457613469205afc"}
Oct 31 16:30:13.307592 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.307575    4417 scope.go:122] "RemoveContainer" containerID="10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"
Oct 31 16:30:13.307919 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:13.307893    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600\": container with ID starting with 10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600 not found: ID does not exist" containerID="10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"
Oct 31 16:30:13.308038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.307928    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600"} err="failed to get container status \"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600\": rpc error: code = NotFound desc = could not find container \"10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600\": container with ID starting with 10178589dfc3501613fca2258828c5dd599b80c570744a65ea544983080f0600 not found: ID does not exist"
Oct 31 16:30:13.308038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.307952    4417 scope.go:122] "RemoveContainer" containerID="ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"
Oct 31 16:30:13.324849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.318628    4417 scope.go:122] "RemoveContainer" containerID="ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"
Oct 31 16:30:13.324849 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:13.319000    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b\": container with ID starting with ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b not found: ID does not exist" containerID="ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"
Oct 31 16:30:13.324849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.319032    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b"} err="failed to get container status \"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b\": rpc error: code = NotFound desc = could not find container \"ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b\": container with ID starting with ea246a286b38184f65fec66475af68f85de8f4abc2464f54416e9bb142c2c01b not found: ID does not exist"
Oct 31 16:30:13.324849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.319050    4417 scope.go:122] "RemoveContainer" containerID="50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"
Oct 31 16:30:13.328325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.328306    4417 scope.go:122] "RemoveContainer" containerID="50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"
Oct 31 16:30:13.328602 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:13.328583    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76\": container with ID starting with 50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76 not found: ID does not exist" containerID="50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"
Oct 31 16:30:13.328703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.328610    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76"} err="failed to get container status \"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76\": rpc error: code = NotFound desc = could not find container \"50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76\": container with ID starting with 50ab7462f49c44e3f3ffc20a32fcf8f9236a1859ac483dc0ea880b2b194add76 not found: ID does not exist"
Oct 31 16:30:13.449138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.449101    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:30:13.494761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.494725    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-lj74w"]
Oct 31 16:30:13.569822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.569720    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:30:13.584668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.584559    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-zx64r"]
Oct 31 16:30:13.710084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.710011    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:30:13.734416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:13.730605    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-xxlfm"]
Oct 31 16:30:14.173412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:14.173383    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:30:15.085280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.085249    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" path="/var/lib/kubelet/pods/4ad1a287-7d7b-4979-ab94-124d1739061a/volumes"
Oct 31 16:30:15.085973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.085953    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" path="/var/lib/kubelet/pods/77980ac3-7ec2-475b-8191-a6988512ae66/volumes"
Oct 31 16:30:15.086531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.086515    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" path="/var/lib/kubelet/pods/d4af8964-24c5-41e7-b001-b5e65fe721e3/volumes"
Oct 31 16:30:15.098526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.098488    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:30:15.184998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.184957    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185012    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185041    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185128    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185150    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185207    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185318    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185374    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.185425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.185404    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\") pod \"31cec1a0-366f-4a49-bea0-d84997b68680\" (UID: \"31cec1a0-366f-4a49-bea0-d84997b68680\") "
Oct 31 16:30:15.186304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.186250    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.186656    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.186771    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.186960    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.187023    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31cec1a0-366f-4a49-bea0-d84997b68680-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.187028    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.187045    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.188661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.188292    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "kube-api-access-2gk7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.189109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.189085    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.190005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.189723    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.190331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.190296    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy" pod "31cec1a0-366f-4a49-bea0-d84997b68680" (UID: "31cec1a0-366f-4a49-bea0-d84997b68680"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.192031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.192007    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:30:15.197360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.197334    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:30:15.200504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.200484    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:30:15.287790 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.287752    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.287987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.287833    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.287987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.287903    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.287987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.287930    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.287987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.287959    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288038    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288092    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288198    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288252    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288305    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288342    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.288389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288373    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.288594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288407    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.288594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288447    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.288594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288497    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\") pod \"af122eb5-87f2-409c-b18c-80b940adb05f\" (UID: \"af122eb5-87f2-409c-b18c-80b940adb05f\") "
Oct 31 16:30:15.288594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288538    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.288594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288591    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.289026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.288646    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\") pod \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\" (UID: \"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a\") "
Oct 31 16:30:15.289417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289393    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31cec1a0-366f-4a49-bea0-d84997b68680-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289421    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289439    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gk7b\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-kube-api-access-2gk7b\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289454    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289467    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289481    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31cec1a0-366f-4a49-bea0-d84997b68680-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289493    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31cec1a0-366f-4a49-bea0-d84997b68680-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.289530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289393    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.289966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289731    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.289966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289731    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.290074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289995    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.290074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.290048    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:15.290212 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.289401    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.290297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.290228    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.290573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.290551    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.290746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.290721    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.291013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.290976    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.291537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.291510    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.293539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.293487    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.293710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.293681    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:30:15.294060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.294034    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:15.294249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.294230    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "kube-api-access-qvlq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.294430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.294411    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token" pod "af122eb5-87f2-409c-b18c-80b940adb05f" (UID: "af122eb5-87f2-409c-b18c-80b940adb05f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.295353 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.295330    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "kube-api-access-4pzgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.295471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.295455    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token" pod "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" (UID: "9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:15.371072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.371012    4417 generic.go:356] "Generic (PLEG): container finished" podID="31cec1a0-366f-4a49-bea0-d84997b68680" containerID="df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843" exitCode=0
Oct 31 16:30:15.371262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.371236    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" event={"ID":"31cec1a0-366f-4a49-bea0-d84997b68680","Type":"ContainerDied","Data":"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"}
Oct 31 16:30:15.371342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.371321    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h" event={"ID":"31cec1a0-366f-4a49-bea0-d84997b68680","Type":"ContainerDied","Data":"fc8398a550dc7da3435850ef160cee78bdac9d91f799f86c8083a5061818e9e0"}
Oct 31 16:30:15.371398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.371358    4417 scope.go:122] "RemoveContainer" containerID="df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"
Oct 31 16:30:15.372560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.372079    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"
Oct 31 16:30:15.385679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.383648    4417 generic.go:356] "Generic (PLEG): container finished" podID="af122eb5-87f2-409c-b18c-80b940adb05f" containerID="9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d" exitCode=0
Oct 31 16:30:15.385679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.383824    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" event={"ID":"af122eb5-87f2-409c-b18c-80b940adb05f","Type":"ContainerDied","Data":"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"}
Oct 31 16:30:15.385679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.383881    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj" event={"ID":"af122eb5-87f2-409c-b18c-80b940adb05f","Type":"ContainerDied","Data":"36efa292232f03f768dda9f5fbadb0e0a24182f7fb17265dfb922120b5660a69"}
Oct 31 16:30:15.385679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.383984    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391371    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391402    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391419    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391432    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391447    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391460    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391473    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391486    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzgw\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-kube-api-access-4pzgw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391502    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391515    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af122eb5-87f2-409c-b18c-80b940adb05f-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391528    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391541    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391554    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af122eb5-87f2-409c-b18c-80b940adb05f-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391575    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391587    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391601    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391645    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af122eb5-87f2-409c-b18c-80b940adb05f-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.392052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.391659    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlq6\" (UniqueName: \"kubernetes.io/projected/af122eb5-87f2-409c-b18c-80b940adb05f-kube-api-access-qvlq6\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:15.394814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.394782    4417 generic.go:356] "Generic (PLEG): container finished" podID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerID="fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6" exitCode=0
Oct 31 16:30:15.394992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.394861    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" event={"ID":"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a","Type":"ContainerDied","Data":"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"}
Oct 31 16:30:15.394992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.394906    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt" event={"ID":"9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a","Type":"ContainerDied","Data":"c479f255835d2300f2eccddf94261b4f993f29847bc8c739baa797cab4c5af39"}
Oct 31 16:30:15.394992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.394983    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"
Oct 31 16:30:15.407208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.407193    4417 scope.go:122] "RemoveContainer" containerID="df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"
Oct 31 16:30:15.407637 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:15.407592    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843\": container with ID starting with df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843 not found: ID does not exist" containerID="df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"
Oct 31 16:30:15.407761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.407742    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843"} err="failed to get container status \"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843\": rpc error: code = NotFound desc = could not find container \"df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843\": container with ID starting with df95f7d739f2772658aa87ae2b236a054f488b994d8f0edba72b00b72f4e3843 not found: ID does not exist"
Oct 31 16:30:15.408088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.407920    4417 scope.go:122] "RemoveContainer" containerID="9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"
Oct 31 16:30:15.420099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.420082    4417 scope.go:122] "RemoveContainer" containerID="9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"
Oct 31 16:30:15.420420 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:15.420401    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d\": container with ID starting with 9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d not found: ID does not exist" containerID="9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"
Oct 31 16:30:15.420545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.420454    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d"} err="failed to get container status \"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d\": rpc error: code = NotFound desc = could not find container \"9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d\": container with ID starting with 9dad7e9a4f46110e8ca17ce6d246e69b5b6fdcc112dd814060df9f1d5c66d95d not found: ID does not exist"
Oct 31 16:30:15.420545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.420471    4417 scope.go:122] "RemoveContainer" containerID="fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"
Oct 31 16:30:15.467901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.466826    4417 scope.go:122] "RemoveContainer" containerID="fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"
Oct 31 16:30:15.467901 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:15.467325    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6\": container with ID starting with fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6 not found: ID does not exist" containerID="fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"
Oct 31 16:30:15.467901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.467364    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6"} err="failed to get container status \"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6\": rpc error: code = NotFound desc = could not find container \"fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6\": container with ID starting with fe0945662812098e933006575a2f25474df0b991f2e3734dfcd1eaa596d492c6 not found: ID does not exist"
Oct 31 16:30:15.691343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.691256    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:30:15.695670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.695638    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-v5q8h"]
Oct 31 16:30:15.822386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.822348    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:30:15.829647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.829555    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-r9rlj"]
Oct 31 16:30:15.924108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.924070    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:30:15.927682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:15.927655    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-h7pbt"]
Oct 31 16:30:16.050693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:16.042562    4417 scope.go:122] "RemoveContainer" containerID="ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"
Oct 31 16:30:16.067021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:16.066980    4417 scope.go:122] "RemoveContainer" containerID="e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"
Oct 31 16:30:17.042394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.042311    4417 scope.go:122] "RemoveContainer" containerID="9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"
Oct 31 16:30:17.070657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.070591    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" path="/var/lib/kubelet/pods/31cec1a0-366f-4a49-bea0-d84997b68680/volumes"
Oct 31 16:30:17.071200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.071178    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" path="/var/lib/kubelet/pods/9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a/volumes"
Oct 31 16:30:17.071715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.071697    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" path="/var/lib/kubelet/pods/af122eb5-87f2-409c-b18c-80b940adb05f/volumes"
Oct 31 16:30:17.413221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.413188    4417 generic.go:356] "Generic (PLEG): container finished" podID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerID="23435007c911743be15d7b898bb9824d20454836184cdf13ea486df0292eb730" exitCode=60
Oct 31 16:30:17.413405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.413261    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerDied","Data":"23435007c911743be15d7b898bb9824d20454836184cdf13ea486df0292eb730"}
Oct 31 16:30:17.413405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.413317    4417 scope.go:122] "RemoveContainer" containerID="e38073994b24539b631c44f0bf3726fbcd4cf30ea202411959a9ebcefc831ecd"
Oct 31 16:30:17.413759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.413729    4417 scope.go:122] "RemoveContainer" containerID="23435007c911743be15d7b898bb9824d20454836184cdf13ea486df0292eb730"
Oct 31 16:30:17.414249 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:17.414159    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl_odh-ai-gateway-infra(4db21af1-60b6-47fc-9647-c8740b6a90b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0"
Oct 31 16:30:17.416098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.416080    4417 generic.go:356] "Generic (PLEG): container finished" podID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerID="aacac8330f9efff209f16b3ba2b00f08b4b5c8c25d22803ef783ea5f93306d26" exitCode=60
Oct 31 16:30:17.416179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.416113    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerDied","Data":"aacac8330f9efff209f16b3ba2b00f08b4b5c8c25d22803ef783ea5f93306d26"}
Oct 31 16:30:17.416475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.416462    4417 scope.go:122] "RemoveContainer" containerID="aacac8330f9efff209f16b3ba2b00f08b4b5c8c25d22803ef783ea5f93306d26"
Oct 31 16:30:17.416768 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:17.416753    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891070-9nxk5_odh-ai-gateway-infra(24e04d90-7091-440f-a935-7cad4db4ddd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8"
Oct 31 16:30:17.426249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:17.426225    4417 scope.go:122] "RemoveContainer" containerID="ae9b87018d83d8faac28b37fe1f2a45adff1516cd614943d6e04422bcfd22dea"
Oct 31 16:30:18.044394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.044356    4417 scope.go:122] "RemoveContainer" containerID="56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"
Oct 31 16:30:18.065670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.065644    4417 scope.go:122] "RemoveContainer" containerID="4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a"
Oct 31 16:30:18.428357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.428273    4417 generic.go:356] "Generic (PLEG): container finished" podID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerID="bf6f09fcef094f7d9df52f2c1786664bbf2614c0dffc79eed1fb4ad615a56fa3" exitCode=60
Oct 31 16:30:18.428357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.428344    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerDied","Data":"bf6f09fcef094f7d9df52f2c1786664bbf2614c0dffc79eed1fb4ad615a56fa3"}
Oct 31 16:30:18.428558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.428415    4417 scope.go:122] "RemoveContainer" containerID="9b06206587a9e9dcb50bce838ca1bc76d732d10f1c3656aa51170fd94387bb69"
Oct 31 16:30:18.428844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.428828    4417 scope.go:122] "RemoveContainer" containerID="bf6f09fcef094f7d9df52f2c1786664bbf2614c0dffc79eed1fb4ad615a56fa3"
Oct 31 16:30:18.429160 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:18.429142    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs_odh-ai-gateway-infra(96f14092-1638-4fd1-98c3-54c8ebf2e8c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8"
Oct 31 16:30:18.529541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.529512    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:18.673287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.672646    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:18.783378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.783130    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:18.899460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.899422    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\" (UniqueName: \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\") pod \"4db21af1-60b6-47fc-9647-c8740b6a90b0\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") "
Oct 31 16:30:18.899678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.899526    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\") pod \"4db21af1-60b6-47fc-9647-c8740b6a90b0\" (UID: \"4db21af1-60b6-47fc-9647-c8740b6a90b0\") "
Oct 31 16:30:18.900275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.900249    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca" pod "4db21af1-60b6-47fc-9647-c8740b6a90b0" (UID: "4db21af1-60b6-47fc-9647-c8740b6a90b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:18.903247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.903225    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5" pod "4db21af1-60b6-47fc-9647-c8740b6a90b0" (UID: "4db21af1-60b6-47fc-9647-c8740b6a90b0"). InnerVolumeSpecName "kube-api-access-lxqt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:18.913371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:18.913346    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:19.001683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.001571    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\") pod \"24e04d90-7091-440f-a935-7cad4db4ddd8\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") "
Oct 31 16:30:19.001872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.001700    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\" (UniqueName: \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\") pod \"24e04d90-7091-440f-a935-7cad4db4ddd8\" (UID: \"24e04d90-7091-440f-a935-7cad4db4ddd8\") "
Oct 31 16:30:19.002044 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.002019    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca" pod "24e04d90-7091-440f-a935-7cad4db4ddd8" (UID: "24e04d90-7091-440f-a935-7cad4db4ddd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:19.002455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.002434    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqt5\" (UniqueName: \"kubernetes.io/projected/4db21af1-60b6-47fc-9647-c8740b6a90b0-kube-api-access-lxqt5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.002601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.002585    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db21af1-60b6-47fc-9647-c8740b6a90b0-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.002703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.002604    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24e04d90-7091-440f-a935-7cad4db4ddd8-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.003916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.003894    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42" pod "24e04d90-7091-440f-a935-7cad4db4ddd8" (UID: "24e04d90-7091-440f-a935-7cad4db4ddd8"). InnerVolumeSpecName "kube-api-access-5ht42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.105402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.105363    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ht42\" (UniqueName: \"kubernetes.io/projected/24e04d90-7091-440f-a935-7cad4db4ddd8-kube-api-access-5ht42\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.487350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.487259    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5" event={"ID":"24e04d90-7091-440f-a935-7cad4db4ddd8","Type":"ContainerDied","Data":"1c4e226cc30a8e285ba93bd38b1e253e856e2d6e28eca559a763656748aecc1b"}
Oct 31 16:30:19.487547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.487369    4417 scope.go:122] "RemoveContainer" containerID="aacac8330f9efff209f16b3ba2b00f08b4b5c8c25d22803ef783ea5f93306d26"
Oct 31 16:30:19.487547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.487537    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"
Oct 31 16:30:19.494541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.494515    4417 generic.go:356] "Generic (PLEG): container finished" podID="7528865b-c72b-4b18-b770-853e4284d4c9" containerID="cfd94a2bfa1b4fb334aeacb5c7a0362da43b0aa4b0af94c862f06b975d765eb4" exitCode=60
Oct 31 16:30:19.494910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.494881    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerDied","Data":"cfd94a2bfa1b4fb334aeacb5c7a0362da43b0aa4b0af94c862f06b975d765eb4"}
Oct 31 16:30:19.495565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.495544    4417 scope.go:122] "RemoveContainer" containerID="cfd94a2bfa1b4fb334aeacb5c7a0362da43b0aa4b0af94c862f06b975d765eb4"
Oct 31 16:30:19.497108 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:19.496541    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2_odh-ai-gateway-infra(7528865b-c72b-4b18-b770-853e4284d4c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" podUID="7528865b-c72b-4b18-b770-853e4284d4c9"
Oct 31 16:30:19.501930 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.501895    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerStarted","Data":"031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e"}
Oct 31 16:30:19.502361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.502340    4417 scope.go:122] "RemoveContainer" containerID="031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e"
Oct 31 16:30:19.502741 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:19.502713    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2_odh-ai-gateway-infra(d37126e0-3210-4d6e-a389-dac6e36aacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff"
Oct 31 16:30:19.504490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.504471    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl" event={"ID":"4db21af1-60b6-47fc-9647-c8740b6a90b0","Type":"ContainerDied","Data":"a577f0dad483258290d3b69ab20cac94741391d3744b118003fe277be75b8ee1"}
Oct 31 16:30:19.504611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.504490    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"
Oct 31 16:30:19.517491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.517471    4417 scope.go:122] "RemoveContainer" containerID="56ffab9fabff6ec7f96aec73826e37d0a5c5499cb4aaffef42cad31fe6781f0b"
Oct 31 16:30:19.533361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.533339    4417 scope.go:122] "RemoveContainer" containerID="23435007c911743be15d7b898bb9824d20454836184cdf13ea486df0292eb730"
Oct 31 16:30:19.571783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.570939    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:19.674274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.673907    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:19.678128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.677475    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891070-9nxk5"]
Oct 31 16:30:19.771888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.771821    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:19.920324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.920258    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\" (UniqueName: \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\") pod \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") "
Oct 31 16:30:19.920512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.920455    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\") pod \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\" (UID: \"96f14092-1638-4fd1-98c3-54c8ebf2e8c8\") "
Oct 31 16:30:19.920854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.920828    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca" pod "96f14092-1638-4fd1-98c3-54c8ebf2e8c8" (UID: "96f14092-1638-4fd1-98c3-54c8ebf2e8c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:19.921047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.921031    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:19.922332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.922307    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf" pod "96f14092-1638-4fd1-98c3-54c8ebf2e8c8" (UID: "96f14092-1638-4fd1-98c3-54c8ebf2e8c8"). InnerVolumeSpecName "kube-api-access-6vwqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:19.989760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.989702    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:19.991932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:19.991892    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc67dd-29891070-dv4pl"]
Oct 31 16:30:20.021938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.021852    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwqf\" (UniqueName: \"kubernetes.io/projected/96f14092-1638-4fd1-98c3-54c8ebf2e8c8-kube-api-access-6vwqf\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:20.126670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.126636    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:30:20.128856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128830    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.128856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128852    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128862    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128867    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128877    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128882    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128891    4417 cpu_manager.go:414] "containerMap: removing container" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128896    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128902    4417 cpu_manager.go:414] "containerMap: removing container" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128907    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128915    4417 cpu_manager.go:414] "containerMap: removing container" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128919    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128929    4417 cpu_manager.go:414] "containerMap: removing container" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128933    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128940    4417 cpu_manager.go:414] "containerMap: removing container" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128944    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128951    4417 cpu_manager.go:414] "containerMap: removing container" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128956    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128962    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128967    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128974    4417 cpu_manager.go:414] "containerMap: removing container" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128980    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128988    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128992    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.128999    4417 cpu_manager.go:414] "containerMap: removing container" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy"
Oct 31 16:30:20.129019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129004    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129099    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129107    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="77980ac3-7ec2-475b-8191-a6988512ae66" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129113    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129121    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="31cec1a0-366f-4a49-bea0-d84997b68680" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129128    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129133    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129139    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129144    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4af8964-24c5-41e7-b001-b5e65fe721e3" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129150    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129155    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad1a287-7d7b-4979-ab94-124d1739061a" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129160    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="af122eb5-87f2-409c-b18c-80b940adb05f" containerName="istio-proxy"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129170    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:20.129756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.129176    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5d3d3f-6625-4f72-9bec-7d8d1bb6127a" containerName="istio-proxy"
Oct 31 16:30:20.133254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.133238    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.184492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.184457    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-6b1cca-serving-cert\""
Oct 31 16:30:20.188474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.188409    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:30:20.326336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.326298    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.326552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.326355    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.326552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.326511    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzr8\" (UniqueName: \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.326814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.326794    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.472179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.465945    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.496199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.496169    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.496516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.496494    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.499083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.498709    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.499083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.498840    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzr8\" (UniqueName: \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.508216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.508184    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.519079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.519053    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.522521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.521808    4417 generic.go:356] "Generic (PLEG): container finished" podID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerID="031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e" exitCode=60
Oct 31 16:30:20.522521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.521928    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerDied","Data":"031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e"}
Oct 31 16:30:20.522521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.521991    4417 scope.go:122] "RemoveContainer" containerID="4e4d30427605ef0bdaf54a97f0c058322ea2cc2ebd607b58f5c007f281f1971a"
Oct 31 16:30:20.522521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.522500    4417 scope.go:122] "RemoveContainer" containerID="031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e"
Oct 31 16:30:20.522956 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:20.522927    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2_odh-ai-gateway-infra(d37126e0-3210-4d6e-a389-dac6e36aacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff"
Oct 31 16:30:20.530509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.530475    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs" event={"ID":"96f14092-1638-4fd1-98c3-54c8ebf2e8c8","Type":"ContainerDied","Data":"c3cb2ee2ed06bb57c989a406acb4261c6a0ee7116af87531c52da9567e71282b"}
Oct 31 16:30:20.530509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.530496    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"
Oct 31 16:30:20.605174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.605097    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzr8\" (UniqueName: \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\") pod \"maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.650038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.650009    4417 scope.go:122] "RemoveContainer" containerID="bf6f09fcef094f7d9df52f2c1786664bbf2614c0dffc79eed1fb4ad615a56fa3"
Oct 31 16:30:20.751575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.751539    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:20.781889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.781855    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:20.789597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.789422    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d04acd-29891070-zfbbs"]
Oct 31 16:30:20.792760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.792733    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:20.889658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:20.889241    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:21.050978 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.050812    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:21.053250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.053220    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" path="/var/lib/kubelet/pods/24e04d90-7091-440f-a935-7cad4db4ddd8/volumes"
Oct 31 16:30:21.053714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.053691    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" path="/var/lib/kubelet/pods/4db21af1-60b6-47fc-9647-c8740b6a90b0/volumes"
Oct 31 16:30:21.054148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.054132    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f14092-1638-4fd1-98c3-54c8ebf2e8c8" path="/var/lib/kubelet/pods/96f14092-1638-4fd1-98c3-54c8ebf2e8c8/volumes"
Oct 31 16:30:21.119440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.119401    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\") pod \"7528865b-c72b-4b18-b770-853e4284d4c9\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") "
Oct 31 16:30:21.119663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.119606    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\" (UniqueName: \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\") pod \"7528865b-c72b-4b18-b770-853e4284d4c9\" (UID: \"7528865b-c72b-4b18-b770-853e4284d4c9\") "
Oct 31 16:30:21.120690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.120665    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca" pod "7528865b-c72b-4b18-b770-853e4284d4c9" (UID: "7528865b-c72b-4b18-b770-853e4284d4c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:21.123214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.123180    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb" pod "7528865b-c72b-4b18-b770-853e4284d4c9" (UID: "7528865b-c72b-4b18-b770-853e4284d4c9"). InnerVolumeSpecName "kube-api-access-xd8lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:21.135892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.133964    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:30:21.138724 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:21.138699    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d968dd0_128a_40a9_a180_23b9d1575526.slice/crio-e37ca1f918c5ff9ac95c23686504e108b3b92adf40db1b7d1676abff334d78d4 WatchSource:0}: Error finding container e37ca1f918c5ff9ac95c23686504e108b3b92adf40db1b7d1676abff334d78d4: Status 404 returned error can't find the container with id e37ca1f918c5ff9ac95c23686504e108b3b92adf40db1b7d1676abff334d78d4
Oct 31 16:30:21.221540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.221502    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7528865b-c72b-4b18-b770-853e4284d4c9-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:21.221540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.221537    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8lb\" (UniqueName: \"kubernetes.io/projected/7528865b-c72b-4b18-b770-853e4284d4c9-kube-api-access-xd8lb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:21.540441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.540406    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" event={"ID":"0d968dd0-128a-40a9-a180-23b9d1575526","Type":"ContainerStarted","Data":"e37ca1f918c5ff9ac95c23686504e108b3b92adf40db1b7d1676abff334d78d4"}
Oct 31 16:30:21.543244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.543215    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2" event={"ID":"7528865b-c72b-4b18-b770-853e4284d4c9","Type":"ContainerDied","Data":"5ba2eebe160aca4db353f9fce7c4fd6721fe09328d2a7f688bfdebe95b516ad5"}
Oct 31 16:30:21.543366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.543246    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"
Oct 31 16:30:21.543366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.543279    4417 scope.go:122] "RemoveContainer" containerID="cfd94a2bfa1b4fb334aeacb5c7a0362da43b0aa4b0af94c862f06b975d765eb4"
Oct 31 16:30:21.713416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.713390    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:21.747442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.747363    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:21.761560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.761513    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4fe3a4-29891070-4d2t2"]
Oct 31 16:30:21.830472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.830434    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\") pod \"d37126e0-3210-4d6e-a389-dac6e36aacff\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") "
Oct 31 16:30:21.830714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.830700    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\" (UniqueName: \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\") pod \"d37126e0-3210-4d6e-a389-dac6e36aacff\" (UID: \"d37126e0-3210-4d6e-a389-dac6e36aacff\") "
Oct 31 16:30:21.830906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.830876    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca" pod "d37126e0-3210-4d6e-a389-dac6e36aacff" (UID: "d37126e0-3210-4d6e-a389-dac6e36aacff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:30:21.831270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.831246    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d37126e0-3210-4d6e-a389-dac6e36aacff-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:21.832897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.832864    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg" pod "d37126e0-3210-4d6e-a389-dac6e36aacff" (UID: "d37126e0-3210-4d6e-a389-dac6e36aacff"). InnerVolumeSpecName "kube-api-access-jzhzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:21.932359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:21.932326    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhzg\" (UniqueName: \"kubernetes.io/projected/d37126e0-3210-4d6e-a389-dac6e36aacff-kube-api-access-jzhzg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:30:22.552679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.552647    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"
Oct 31 16:30:22.553132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.552643    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2" event={"ID":"d37126e0-3210-4d6e-a389-dac6e36aacff","Type":"ContainerDied","Data":"1c9cba3921e597fc5e399c38bef1466278667f5cce781cc5370a42be0883ae85"}
Oct 31 16:30:22.553132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.552797    4417 scope.go:122] "RemoveContainer" containerID="031a48b45e7cbf237b82331451113a050ff05944f8633e7df9bc055c1c362b5e"
Oct 31 16:30:22.554590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.554570    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" event={"ID":"0d968dd0-128a-40a9-a180-23b9d1575526","Type":"ContainerStarted","Data":"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"}
Oct 31 16:30:22.554814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.554796    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:22.712195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.712139    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" podStartSLOduration=1.814186185 podStartE2EDuration="2.712122905s" totalImagesPullingTime="897.93672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:22.706897915 +0000 UTC m=+1667.768473101" watchObservedRunningTime="2026-10-31 16:30:22.712122905 +0000 UTC m=+1667.773698085"
Oct 31 16:30:22.809962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.809876    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:22.814536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.814222    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7f13a-29891070-mxql2"]
Oct 31 16:30:22.937413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.937365    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:30:22.937589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:22.937438    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:30:23.052035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:23.051991    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" path="/var/lib/kubelet/pods/7528865b-c72b-4b18-b770-853e4284d4c9/volumes"
Oct 31 16:30:23.052494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:23.052473    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" path="/var/lib/kubelet/pods/d37126e0-3210-4d6e-a389-dac6e36aacff/volumes"
Oct 31 16:30:24.573292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.573259    4417 generic.go:356] "Generic (PLEG): container finished" podID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerID="4dca61d2db839764a81fbdc2d93aca2dd32309850535195b121f227d0c1a2817" exitCode=0
Oct 31 16:30:24.573914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.573336    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerDied","Data":"4dca61d2db839764a81fbdc2d93aca2dd32309850535195b121f227d0c1a2817"}
Oct 31 16:30:24.573996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.573938    4417 scope.go:122] "RemoveContainer" containerID="4dca61d2db839764a81fbdc2d93aca2dd32309850535195b121f227d0c1a2817"
Oct 31 16:30:24.575751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.575721    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:24.575849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.575794    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:24.575849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:24.575825    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:25.583666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:25.583596    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerStarted","Data":"5e1c74bbc2325d7e3010ec4676d9dd7c3ada844f9f54742196f8d81d3f3a7dad"}
Oct 31 16:30:26.301984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.301954    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:30:26.303327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303301    4417 cpu_manager.go:414] "containerMap: removing container" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:26.303327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303323    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303338    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303343    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303362    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303367    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303392    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303396    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303404    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303408    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303419    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303423    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303432    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303436    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303528    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db21af1-60b6-47fc-9647-c8740b6a90b0" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303538    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303546    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303552    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37126e0-3210-4d6e-a389-dac6e36aacff" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303557    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303563    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e04d90-7091-440f-a935-7cad4db4ddd8" containerName="cleanup"
Oct 31 16:30:26.303870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.303573    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:26.307218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.307194    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.361019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.360841    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-822dd4-serving-cert\""
Oct 31 16:30:26.410744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.410708    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:30:26.420630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.420586    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.420791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.420642    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.420791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.420741    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.420791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.420773    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgpp\" (UniqueName: \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.522559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.522519    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.522559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.522561    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.522825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.522608    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.522825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.522641    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgpp\" (UniqueName: \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.522825 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:26.522763    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-serving-cert: secret "maas-api-e2e-shared-a-w4-822dd4-serving-cert" not found
Oct 31 16:30:26.523009 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:26.522849    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls podName:bdfc31c6-2c77-4d27-b70c-d20ea6c4946a nodeName:}" failed. No retries permitted until 2026-10-31 16:30:27.022831823 +0000 UTC m=+1672.084406991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls") pod "maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" (UID: "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a") : secret "maas-api-e2e-shared-a-w4-822dd4-serving-cert" not found
Oct 31 16:30:26.523330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.523307    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.525133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.525106    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:26.566691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.565909    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:30:26.566691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.566042    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:30:26.566691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.566082    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 16:30:26.605684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:26.603925    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgpp\" (UniqueName: \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:27.063642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.063495    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:27.067768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.067560    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:27.223151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.223106    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:27.565126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.565081    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:30:27.565307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.565150    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 16:30:27.625199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.625170    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:30:27.628220 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:27.628190    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfc31c6_2c77_4d27_b70c_d20ea6c4946a.slice/crio-e2423b02cce2b4e88f4832d9fe2d39e994ae92840b423b7d43170511a16ff444 WatchSource:0}: Error finding container e2423b02cce2b4e88f4832d9fe2d39e994ae92840b423b7d43170511a16ff444: Status 404 returned error can't find the container with id e2423b02cce2b4e88f4832d9fe2d39e994ae92840b423b7d43170511a16ff444
Oct 31 16:30:27.679673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:27.679612    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" event={"ID":"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a","Type":"ContainerStarted","Data":"e2423b02cce2b4e88f4832d9fe2d39e994ae92840b423b7d43170511a16ff444"}
Oct 31 16:30:28.574713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.568786    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:30:28.574713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.568875    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 16:30:28.600440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.600411    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:30:28.688185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.688152    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" event={"ID":"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a","Type":"ContainerStarted","Data":"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"}
Oct 31 16:30:28.688528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.688245    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:28.828917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:28.828853    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" podStartSLOduration=1.9641336740000002 podStartE2EDuration="2.828829581s" totalImagesPullingTime="864.695907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:28.827361097 +0000 UTC m=+1673.888936283" watchObservedRunningTime="2026-10-31 16:30:28.828829581 +0000 UTC m=+1673.890404766"
Oct 31 16:30:29.568242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:29.568203    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:30:29.568561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:29.568521    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 16:30:30.565182 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:30.565139    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 16:30:30.565775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:30.565208    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 16:30:31.565130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.565086    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:30:31.566876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.566846    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:31.566876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.566875    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:31.567375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.567088    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7528865b-c72b-4b18-b770-853e4284d4c9" containerName="cleanup"
Oct 31 16:30:31.570887 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.570865    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.571897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.571875    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:30:31.572198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.572176    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:30:31.573067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.573048    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:30:31.621303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.621274    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dac4fa-serving-cert\""
Oct 31 16:30:31.656260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.656224    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg92h\" (UniqueName: \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.656473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.656287    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.656473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.656406    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.656729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.656703    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.672916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.672881    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:30:31.758385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.758340    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg92h\" (UniqueName: \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.758599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.758398    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.758599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.758476    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.758599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.758556    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.758855 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:31.758697    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-serving-cert: secret "maas-api-e2e-api-a-dac4fa-serving-cert" not found
Oct 31 16:30:31.758855 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:31.758779    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls podName:8d2dfe63-3572-49c6-86af-6316c24470b7 nodeName:}" failed. No retries permitted until 2026-10-31 16:30:32.25875005 +0000 UTC m=+1677.320325231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls") pod "maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" (UID: "8d2dfe63-3572-49c6-86af-6316c24470b7") : secret "maas-api-e2e-api-a-dac4fa-serving-cert" not found
Oct 31 16:30:31.759358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.759330    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.761282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.761254    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:31.815098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:31.813241    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg92h\" (UniqueName: \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:32.269514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:32.269469    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:32.272249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:32.272220    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") pod \"maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:32.487190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:32.487157    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:32.863310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:32.863218    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:30:32.878740 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:32.878700    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2dfe63_3572_49c6_86af_6316c24470b7.slice/crio-50a2c5bc64eec3fb72c62a49546032aae1b824ae81ae5ec946ba91e3ba63c421 WatchSource:0}: Error finding container 50a2c5bc64eec3fb72c62a49546032aae1b824ae81ae5ec946ba91e3ba63c421: Status 404 returned error can't find the container with id 50a2c5bc64eec3fb72c62a49546032aae1b824ae81ae5ec946ba91e3ba63c421
Oct 31 16:30:32.949199 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:32.949157    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:30:33.774837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:33.774789    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" event={"ID":"8d2dfe63-3572-49c6-86af-6316c24470b7","Type":"ContainerStarted","Data":"50a2c5bc64eec3fb72c62a49546032aae1b824ae81ae5ec946ba91e3ba63c421"}
Oct 31 16:30:34.702133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:34.702102    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:30:34.847641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:34.845833    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" event={"ID":"8d2dfe63-3572-49c6-86af-6316c24470b7","Type":"ContainerStarted","Data":"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"}
Oct 31 16:30:34.847641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:34.847077    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:34.967190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:34.965236    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" podStartSLOduration=3.018727835 podStartE2EDuration="3.965214007s" totalImagesPullingTime="946.486172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:34.961917465 +0000 UTC m=+1680.023492671" watchObservedRunningTime="2026-10-31 16:30:34.965214007 +0000 UTC m=+1680.026789195"
Oct 31 16:30:35.480190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.477461    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:30:35.483109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.483083    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.531354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.531320    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e977aa-openshift-default-dockercfg-7sj8b\""
Oct 31 16:30:35.585501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.585469    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:30:35.597003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.596971    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.597215 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.597021    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.597215 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.597122    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.597398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.597272    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.597398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.597304    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.598091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.598062    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.598190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.598172    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmsz\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.598239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.598224    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.598354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.598338    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.706853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.706803    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmsz\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.707498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.707466    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.707794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.707773    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.707960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.707945    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.708076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.708062    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.708183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.708169    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.708362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.708348    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.708457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.708443    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.708638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.708603    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.714412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.714383    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.716077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.715925    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.716557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.716525    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.717189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.717168    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.717691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.717672    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.722516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.722490    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.723890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.723864    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.770639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.770206    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmsz\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.770639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.770275    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\") pod \"e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") " pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:35.798912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:35.798887    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:36.271938 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:36.271871    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec8e8c4_aa66_4e8e_bfda_f1f07ad3b1be.slice/crio-f0b8f845268a94b6e70a0f6960ff648d11023c1a1abe37c43abf5fd61b33b7a8 WatchSource:0}: Error finding container f0b8f845268a94b6e70a0f6960ff648d11023c1a1abe37c43abf5fd61b33b7a8: Status 404 returned error can't find the container with id f0b8f845268a94b6e70a0f6960ff648d11023c1a1abe37c43abf5fd61b33b7a8
Oct 31 16:30:36.275186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.275144    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:36.275312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.275238    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:36.275312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.275300    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:36.275506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.275482    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:30:36.866011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.865975    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerStarted","Data":"2124dba75bb820e75ee3c83236da1ae89debddc4f0515779aebc8f4d11baef0c"}
Oct 31 16:30:36.866543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.866042    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerStarted","Data":"f0b8f845268a94b6e70a0f6960ff648d11023c1a1abe37c43abf5fd61b33b7a8"}
Oct 31 16:30:36.915156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.915109    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:30:36.920120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.920090    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:36.928507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.928474    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:36.928803 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.928772    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpvv\" (UniqueName: \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:36.928930 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.928908    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:36.929158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.929138    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:36.968091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.968054    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-f43174-serving-cert\""
Oct 31 16:30:36.974809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:36.974779    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:30:37.020847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.020778    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podStartSLOduration=2.020760562 podStartE2EDuration="2.020760562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:37.016510677 +0000 UTC m=+1682.078085863" watchObservedRunningTime="2026-10-31 16:30:37.020760562 +0000 UTC m=+1682.082335750"
Oct 31 16:30:37.030153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.030118    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.030338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.030189    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.030338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.030231    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.030338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.030328    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpvv\" (UniqueName: \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.030515 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:37.030414    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-serving-cert: secret "maas-api-e2e-shared-b-w5-f43174-serving-cert" not found
Oct 31 16:30:37.030515 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:30:37.030479    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls podName:c494b161-faae-4aba-ad28-1606655a9aae nodeName:}" failed. No retries permitted until 2026-10-31 16:30:37.530461533 +0000 UTC m=+1682.592036698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls") pod "maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" (UID: "c494b161-faae-4aba-ad28-1606655a9aae") : secret "maas-api-e2e-shared-b-w5-f43174-serving-cert" not found
Oct 31 16:30:37.030917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.030894    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.032928 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.032906    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.086684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.086651    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpvv\" (UniqueName: \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.548961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.548917    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.551506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.551474    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.799130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.799048    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:37.804958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.804931    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:37.835555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.835528    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:37.874188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.873944    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:37.874862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:37.874842    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:30:38.228855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:38.228824    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:30:38.230225 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:38.230196    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc494b161_faae_4aba_ad28_1606655a9aae.slice/crio-1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9 WatchSource:0}: Error finding container 1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9: Status 404 returned error can't find the container with id 1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9
Oct 31 16:30:38.881133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:38.881089    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" event={"ID":"c494b161-faae-4aba-ad28-1606655a9aae","Type":"ContainerStarted","Data":"1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9"}
Oct 31 16:30:39.889413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:39.889364    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" event={"ID":"c494b161-faae-4aba-ad28-1606655a9aae","Type":"ContainerStarted","Data":"bd92d8125acf7bf10369ac2a301aa9646225a206e6ed237405ec1330846a9cec"}
Oct 31 16:30:39.889874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:39.889640    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:39.996181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:39.996130    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" podStartSLOduration=3.01092717 podStartE2EDuration="3.996115673s" totalImagesPullingTime="985.188503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:39.995094905 +0000 UTC m=+1685.056670090" watchObservedRunningTime="2026-10-31 16:30:39.996115673 +0000 UTC m=+1685.057690898"
Oct 31 16:30:41.872842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:41.872810    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:30:45.810778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:45.789607    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:30:46.160701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.146392    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:30:46.160701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.146595    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:30:46.201639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.165070    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.241660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.239813    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:30:46.255934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.255585    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e977aa\""
Oct 31 16:30:46.255934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.255585    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e977aa-dockercfg-phrm5\""
Oct 31 16:30:46.280642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.256219    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:30:46.280642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.256326    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.299999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.299967    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.301766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.301724    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqc7\" (UniqueName: \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.301938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.301807    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjnh\" (UniqueName: \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.302993 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.302889    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.405721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.405681    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.405921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.405770    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqc7\" (UniqueName: \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.405921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.405799    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjnh\" (UniqueName: \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.406069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.406025    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.406518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.406493    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.406729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.406710    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.465474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.465392    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqc7\" (UniqueName: \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\") pod \"payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:46.470692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.470654    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjnh\" (UniqueName: \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\") pod \"payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.617740 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.617704    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:46.661651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:46.656415    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:47.034168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:47.034139    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:30:47.139322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:47.139293    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:30:47.146170 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:30:47.145467    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5c9200_84b0_4b20_bff8_7a54dc6fcf3b.slice/crio-bcb1cfb02e7953cb6d48cd02feb199d672e34d0b7aa64fe10abf22d2b836f034 WatchSource:0}: Error finding container bcb1cfb02e7953cb6d48cd02feb199d672e34d0b7aa64fe10abf22d2b836f034: Status 404 returned error can't find the container with id bcb1cfb02e7953cb6d48cd02feb199d672e34d0b7aa64fe10abf22d2b836f034
Oct 31 16:30:48.063226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.062136    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" event={"ID":"c1b2d24f-8d3a-4090-85ce-ce1815f100c3","Type":"ContainerStarted","Data":"6f91ac95c0ad72805700e332f909906639c6e09be688dd00a2c38d5a705e96c2"}
Oct 31 16:30:48.063226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.062237    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" event={"ID":"c1b2d24f-8d3a-4090-85ce-ce1815f100c3","Type":"ContainerStarted","Data":"d1ac115cb0d4e590b30f667ae1a4728d635d3c72e7f35ebe720bb0173d4d611e"}
Oct 31 16:30:48.063226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.062929    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:48.065133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.065097    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" event={"ID":"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b","Type":"ContainerStarted","Data":"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"}
Oct 31 16:30:48.065301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.065164    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" event={"ID":"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b","Type":"ContainerStarted","Data":"bcb1cfb02e7953cb6d48cd02feb199d672e34d0b7aa64fe10abf22d2b836f034"}
Oct 31 16:30:48.065301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.065230    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:30:48.177638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.176865    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" podStartSLOduration=3.176846236 podStartE2EDuration="3.176846236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:48.172325255 +0000 UTC m=+1693.233900441" watchObservedRunningTime="2026-10-31 16:30:48.176846236 +0000 UTC m=+1693.238421423"
Oct 31 16:30:48.277201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:48.276997    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" podStartSLOduration=3.2769824610000002 podStartE2EDuration="3.276982461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:30:48.276589302 +0000 UTC m=+1693.338164486" watchObservedRunningTime="2026-10-31 16:30:48.276982461 +0000 UTC m=+1693.338557646"
Oct 31 16:30:51.089105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.089070    4417 generic.go:356] "Generic (PLEG): container finished" podID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerID="c64f3b38258e4595a8f630e3d86197e7080f3fe4c6b27744a089ca20f5dc8a0b" exitCode=0
Oct 31 16:30:51.089667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.089150    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerDied","Data":"c64f3b38258e4595a8f630e3d86197e7080f3fe4c6b27744a089ca20f5dc8a0b"}
Oct 31 16:30:51.089791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.089773    4417 scope.go:122] "RemoveContainer" containerID="c64f3b38258e4595a8f630e3d86197e7080f3fe4c6b27744a089ca20f5dc8a0b"
Oct 31 16:30:51.091452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.091423    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:51.091520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.091496    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:51.091555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:51.091533    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:30:52.099098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:52.099064    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerStarted","Data":"cc14e1c21871cab1051e9461458c4413325bbfda18a1e95bd0ce67c2ea5a3a36"}
Oct 31 16:30:52.920293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:52.920244    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:30:52.920655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:52.920585    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:30:52.920746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:52.920711    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 16:30:53.920238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:53.920204    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:30:53.920648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:53.920261    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 16:30:54.920962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:54.920921    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:30:54.921460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:54.920996    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 16:30:55.920806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:55.920770    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:30:55.921136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:55.920849    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 16:30:56.920482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:56.920444    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 31 16:30:56.920731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:56.920500    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 31 16:30:57.925911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:57.925878    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:30:57.926306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:57.926165    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:30:57.926843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:57.926827    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:30:59.077539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:59.077509    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:30:59.078017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:30:59.077795    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:31:02.946681 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:31:02.946643    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:31:06.636438 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.636399    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:31:06.642516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.642493    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.698810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.698770    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:31:06.738443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.738397    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1b30a15\"/\"openshift-service-ca.crt\""
Oct 31 16:31:06.738712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.738685    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1b30a15\"/\"default-dockercfg-p4g97\""
Oct 31 16:31:06.738841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.738732    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a1b30a15\"/\"e2e-lifecycle-model-a1b30a15-kserve-self-signed-certs\""
Oct 31 16:31:06.738841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.738753    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a1b30a15\"/\"kube-root-ca.crt\""
Oct 31 16:31:06.770291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770247    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.770460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770361    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.770460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770399    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx825\" (UniqueName: \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.770547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770486    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.770547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770535    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.770712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.770690    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872199    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872545    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872600    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx825\" (UniqueName: \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872698    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872774    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.872875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.872860    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.873447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.873426    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.874187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.874161    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.876991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.876964    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.878066 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.878041    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.885069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.884992    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.929541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.929469    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx825\" (UniqueName: \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\") pod \"e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") " pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:06.955548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:06.955512    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:07.382426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:07.380509    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:31:07.385954 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:31:07.385920    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1695e578_deae_4755_9b9f_92707ddb8275.slice/crio-75bd6627b831071de8d0f0bce02b5652a923651f43ee2eb22dd3afb30dbc70dc WatchSource:0}: Error finding container 75bd6627b831071de8d0f0bce02b5652a923651f43ee2eb22dd3afb30dbc70dc: Status 404 returned error can't find the container with id 75bd6627b831071de8d0f0bce02b5652a923651f43ee2eb22dd3afb30dbc70dc
Oct 31 16:31:08.266420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:08.266383    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" event={"ID":"1695e578-deae-4755-9b9f-92707ddb8275","Type":"ContainerStarted","Data":"dc3d108618bb5d486841c8b0de6be7433766970f799e5cd65f4e8d622cb4e1d1"}
Oct 31 16:31:08.266420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:08.266428    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" event={"ID":"1695e578-deae-4755-9b9f-92707ddb8275","Type":"ContainerStarted","Data":"75bd6627b831071de8d0f0bce02b5652a923651f43ee2eb22dd3afb30dbc70dc"}
Oct 31 16:31:08.372244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:08.372180    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" podStartSLOduration=2.372165898 podStartE2EDuration="2.372165898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:08.372093973 +0000 UTC m=+1713.433669158" watchObservedRunningTime="2026-10-31 16:31:08.372165898 +0000 UTC m=+1713.433741083"
Oct 31 16:31:08.955686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:08.955653    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:11.778673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.778639    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:31:11.784257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.783534    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.873062 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.873021    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928cw\" (UniqueName: \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.873244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.873100    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.873244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.873136    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.873244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.873172    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.882345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.882315    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-705a67-serving-cert\""
Oct 31 16:31:11.888774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.888736    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:31:11.974427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.974381    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928cw\" (UniqueName: \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.974668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.974477    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.974668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.974517    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.974668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.974564    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.975368 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:31:11.975329    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-serving-cert: secret "maas-api-e2e-shared-b-w2-705a67-serving-cert" not found
Oct 31 16:31:11.975491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.975420    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:11.975491 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:31:11.975433    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls podName:61214a6c-c042-4c39-b547-429158e529df nodeName:}" failed. No retries permitted until 2026-10-31 16:31:12.475410661 +0000 UTC m=+1717.536985831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls") pod "maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" (UID: "61214a6c-c042-4c39-b547-429158e529df") : secret "maas-api-e2e-shared-b-w2-705a67-serving-cert" not found
Oct 31 16:31:11.977123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:11.977102    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:12.028309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:12.028269    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-928cw\" (UniqueName: \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:12.489732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:12.489694    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:12.492550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:12.492520    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:12.700102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:12.700070    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:13.081481 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:13.081449    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:31:13.083606 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:31:13.083580    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61214a6c_c042_4c39_b547_429158e529df.slice/crio-5fceaed4c228e0ec732e10ac286dd26cce3f22c3c3877def02c93ea60524a11a WatchSource:0}: Error finding container 5fceaed4c228e0ec732e10ac286dd26cce3f22c3c3877def02c93ea60524a11a: Status 404 returned error can't find the container with id 5fceaed4c228e0ec732e10ac286dd26cce3f22c3c3877def02c93ea60524a11a
Oct 31 16:31:13.319361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:13.319327    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" event={"ID":"61214a6c-c042-4c39-b547-429158e529df","Type":"ContainerStarted","Data":"5fceaed4c228e0ec732e10ac286dd26cce3f22c3c3877def02c93ea60524a11a"}
Oct 31 16:31:14.327772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:14.327732    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" event={"ID":"61214a6c-c042-4c39-b547-429158e529df","Type":"ContainerStarted","Data":"ec437d91b985c5faaa88d881c3522bdf72d8027336c4a13b094e430aa08c442b"}
Oct 31 16:31:14.328208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:14.327973    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:14.437361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:14.437305    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" podStartSLOduration=2.493904383 podStartE2EDuration="3.43729051s" totalImagesPullingTime="943.386127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:14.434112289 +0000 UTC m=+1719.495687476" watchObservedRunningTime="2026-10-31 16:31:14.43729051 +0000 UTC m=+1719.498865696"
Oct 31 16:31:16.956290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:16.956246    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:16.969731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:16.969684    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:17.363503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:17.363461    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:31:20.342128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:20.342087    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:31:32.950056 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:31:32.950012    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:31:33.525052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.525020    4417 generic.go:356] "Generic (PLEG): container finished" podID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerID="7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8" exitCode=0
Oct 31 16:31:33.525237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.525094    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerDied","Data":"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"}
Oct 31 16:31:33.525558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.525543    4417 scope.go:122] "RemoveContainer" containerID="7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"
Oct 31 16:31:33.527243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.527216    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:33.527320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.527284    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:33.527320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.527316    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:33.830941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.830903    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:31:33.835991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.835963    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.844179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.844140    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.844485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.844463    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.844707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.844681    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.844906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.844887    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.845053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.845039    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.845455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.845427    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhrj\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.845570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.845512    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.845795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.845596    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.845795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.845709    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.849725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.849700    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:31:33.855767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.855747    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.885888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.885857    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-f43174-openshift-default-dockercfg-9wjgc\""
Oct 31 16:31:33.895098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.895065    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:31:33.948992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.948946    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949065    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949140    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949177    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.949400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949217    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.949400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949288    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.949400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949317    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.949400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949375    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949432    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949486    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc892\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949555    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.949642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949602    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.951669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.949727    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:33.952141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.951683    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhrj\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.951800    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.951839    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.951899    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.952031    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.952495    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.950905    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.952805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.951317    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.953098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.953074    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.955306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.955277    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.979647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.964456    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:31:33.979647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.966463    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:33.988731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:33.988696    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:34.005271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.005230    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhrj\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:34.012931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.012883    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:34.053192 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053154    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053203    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053241    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053318    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053362    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053405    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc892\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053452    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053483    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.053694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053545    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.054019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.053990    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.054019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.054008    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.054385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.054364    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.054551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.054413    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.054691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.054571    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.056882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.056431    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.056882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.056787    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.109866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.109793    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.110004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.109980    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc892\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.155297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.155252    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:34.186853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.186824    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:34.533379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.533296    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerStarted","Data":"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"}
Oct 31 16:31:34.592663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.590930    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:31:34.596651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.596604    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:31:34.678133 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:31:34.674130    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dea52a9_cfc5_434e_812f_bdacf803b5b2.slice/crio-98603ae18814f888ad27ad947d224cc860e16b6920fae791e8638ea690b545fd WatchSource:0}: Error finding container 98603ae18814f888ad27ad947d224cc860e16b6920fae791e8638ea690b545fd: Status 404 returned error can't find the container with id 98603ae18814f888ad27ad947d224cc860e16b6920fae791e8638ea690b545fd
Oct 31 16:31:34.678133 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:31:34.676369    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb1db46_e61a_4fa8_88ef_fffc69a9eb54.slice/crio-15d7ca1fa94978a452acfb145fe6f6d7f0da3bc71b1df9882ab3b7e14a99e574 WatchSource:0}: Error finding container 15d7ca1fa94978a452acfb145fe6f6d7f0da3bc71b1df9882ab3b7e14a99e574: Status 404 returned error can't find the container with id 15d7ca1fa94978a452acfb145fe6f6d7f0da3bc71b1df9882ab3b7e14a99e574
Oct 31 16:31:34.687225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.687068    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.690453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.690408    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.690913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.690504    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.690913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.690549    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.693881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.693816    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:34.693984 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:34.693922    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:31:35.157774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.153525    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:31:35.208225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.208192    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:31:35.210085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.210048    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:35.210233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.210108    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:35.571646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.570684    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" event={"ID":"1dea52a9-cfc5-434e-812f-bdacf803b5b2","Type":"ContainerStarted","Data":"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"}
Oct 31 16:31:35.571646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.570756    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" event={"ID":"1dea52a9-cfc5-434e-812f-bdacf803b5b2","Type":"ContainerStarted","Data":"98603ae18814f888ad27ad947d224cc860e16b6920fae791e8638ea690b545fd"}
Oct 31 16:31:35.576456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.576423    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" event={"ID":"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54","Type":"ContainerStarted","Data":"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"}
Oct 31 16:31:35.576699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.576681    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" event={"ID":"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54","Type":"ContainerStarted","Data":"15d7ca1fa94978a452acfb145fe6f6d7f0da3bc71b1df9882ab3b7e14a99e574"}
Oct 31 16:31:35.682640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.682523    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podStartSLOduration=2.6825029110000003 podStartE2EDuration="2.682502911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:35.679197637 +0000 UTC m=+1740.740772850" watchObservedRunningTime="2026-10-31 16:31:35.682502911 +0000 UTC m=+1740.744078099"
Oct 31 16:31:35.818983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:35.818916    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podStartSLOduration=2.8188971609999998 podStartE2EDuration="2.818897161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:35.818649192 +0000 UTC m=+1740.880224359" watchObservedRunningTime="2026-10-31 16:31:35.818897161 +0000 UTC m=+1740.880472350"
Oct 31 16:31:36.159642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.158687    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:36.159642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.158875    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.159642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.158921    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:36.192833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.192795    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:36.193202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.193174    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.193396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.193369    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:36.209208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.209171    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:36.209507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:36.209475    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:37.156641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.156231    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:37.156641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.156316    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:37.189640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.189104    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:37.189640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.189166    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:37.223825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.212580    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:37.223825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:37.212668    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:38.158645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.155984    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:38.158645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.156044    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:38.206642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.193672    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:38.206642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.193759    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:38.221692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.218727    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:38.221692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:38.218786    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:39.195643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.192740    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:39.195643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.192819    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:39.197828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.197700    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:39.197828 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.197773    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:39.218810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.218688    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:39.218810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:39.218747    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:40.162084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.162043    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:40.162384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.162354    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:40.189922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.189885    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:40.190209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.190177    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:40.207763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.207654    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 16:31:40.207763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:40.207708    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 16:31:41.159800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.159763    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 16:31:41.160401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.160368    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 16:31:41.187672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.187549    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 16:31:41.187672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.187604    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 16:31:41.212309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.212271    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:31:41.216298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:41.216263    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:31:42.161681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.161647    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:42.162163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.161957    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:42.162826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.162808    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:31:42.192471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.192440    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:42.192723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.192704    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:31:42.193448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:31:42.193430    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:32:02.306651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.286914    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:32:02.314961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.314927    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.342666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.342604    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hj7p\" (UniqueName: \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.343107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.342632    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:32:02.343291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.343164    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.470686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.470641    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.491772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.486294    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hj7p\" (UniqueName: \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.535434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.535399    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-487213-dockercfg-9fc4m\""
Oct 31 16:32:02.584324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.535990    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-487213\""
Oct 31 16:32:02.584529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.557194    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.713650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.699679    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hj7p\" (UniqueName: \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\") pod \"payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:02.776236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.776200    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:32:02.819598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.819564    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:32:02.819866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.819848    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:02.820173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.820148    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:32:02.835846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.827744    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:32:02.838200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.833832    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:02.902809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.891818    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e977aa-serving-cert\""
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.927842    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.927901    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.927955    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.927996    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.928025    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:02.939675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.928051    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8rz\" (UniqueName: \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:02.999320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:02.997956    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:03.035115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.034533    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.035530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.035463    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:03.037724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.037418    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.037724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.037455    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:03.037724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.037721    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.041819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.041795    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.042548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.042523    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.042765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.042738    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8rz\" (UniqueName: \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:03.051271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.051215    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.075658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.075060    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.075658 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:32:03.075223    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:32:03.121434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.113122    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8rz\" (UniqueName: \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\") pod \"payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:03.130883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.130841    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\") pod \"maas-api-e2e-api-a-e977aa-d8948f57f-5mg26\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.280079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.279298    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:03.280079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.279670    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:03.595798 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.595766    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:32:03.596517 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:03.596490    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9407fa21_addb_4e36_aaf9_eb984c4f6f5a.slice/crio-3f0cdbe5d96c8e15c9fa49017e8a3500a3d6d71476b8886c79b6bcf04d630cba WatchSource:0}: Error finding container 3f0cdbe5d96c8e15c9fa49017e8a3500a3d6d71476b8886c79b6bcf04d630cba: Status 404 returned error can't find the container with id 3f0cdbe5d96c8e15c9fa49017e8a3500a3d6d71476b8886c79b6bcf04d630cba
Oct 31 16:32:03.708078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.708033    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:32:03.708279 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:03.708256    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abbdee1_7a39_4aaa_b7b6_68b0e322e5f8.slice/crio-8949393c7ae98a0125470e73176b26a5447fdc052fa8c0cf215c65a90497bc2d WatchSource:0}: Error finding container 8949393c7ae98a0125470e73176b26a5447fdc052fa8c0cf215c65a90497bc2d: Status 404 returned error can't find the container with id 8949393c7ae98a0125470e73176b26a5447fdc052fa8c0cf215c65a90497bc2d
Oct 31 16:32:03.757734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:03.757702    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:32:03.760120 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:03.760089    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b1c3cb_a618_4329_8772_774c3d5b110c.slice/crio-98e7795439aadcefea569bfe34204237b10a534fd7bdb0fb4a413d25f2d05c97 WatchSource:0}: Error finding container 98e7795439aadcefea569bfe34204237b10a534fd7bdb0fb4a413d25f2d05c97: Status 404 returned error can't find the container with id 98e7795439aadcefea569bfe34204237b10a534fd7bdb0fb4a413d25f2d05c97
Oct 31 16:32:04.320116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.320064    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" event={"ID":"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8","Type":"ContainerStarted","Data":"7da883ce1d123fd36290de2ab54989d4c2c1d8e3b0b772ac0aac7bbe1e154b2c"}
Oct 31 16:32:04.320301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.320147    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" event={"ID":"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8","Type":"ContainerStarted","Data":"8949393c7ae98a0125470e73176b26a5447fdc052fa8c0cf215c65a90497bc2d"}
Oct 31 16:32:04.320301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.320179    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:04.321507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.321482    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" event={"ID":"f2b1c3cb-a618-4329-8772-774c3d5b110c","Type":"ContainerStarted","Data":"98e7795439aadcefea569bfe34204237b10a534fd7bdb0fb4a413d25f2d05c97"}
Oct 31 16:32:04.323361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.323337    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" event={"ID":"9407fa21-addb-4e36-aaf9-eb984c4f6f5a","Type":"ContainerStarted","Data":"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"}
Oct 31 16:32:04.323498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.323376    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" event={"ID":"9407fa21-addb-4e36-aaf9-eb984c4f6f5a","Type":"ContainerStarted","Data":"3f0cdbe5d96c8e15c9fa49017e8a3500a3d6d71476b8886c79b6bcf04d630cba"}
Oct 31 16:32:04.323498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.323472    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:04.426339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.426283    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" podStartSLOduration=2.4262679240000002 podStartE2EDuration="2.426267924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:04.424610503 +0000 UTC m=+1769.486185689" watchObservedRunningTime="2026-10-31 16:32:04.426267924 +0000 UTC m=+1769.487843110"
Oct 31 16:32:04.530800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:04.530743    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" podStartSLOduration=2.530729328 podStartE2EDuration="2.530729328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:04.528850777 +0000 UTC m=+1769.590425964" watchObservedRunningTime="2026-10-31 16:32:04.530729328 +0000 UTC m=+1769.592304514"
Oct 31 16:32:05.333426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:05.333385    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" event={"ID":"f2b1c3cb-a618-4329-8772-774c3d5b110c","Type":"ContainerStarted","Data":"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"}
Oct 31 16:32:05.333919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:05.333708    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:05.442113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:05.442050    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" podStartSLOduration=2.42403362 podStartE2EDuration="3.442028023s" totalImagesPullingTime="1.017994403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:05.438643877 +0000 UTC m=+1770.500219064" watchObservedRunningTime="2026-10-31 16:32:05.442028023 +0000 UTC m=+1770.503603209"
Oct 31 16:32:11.270964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.270925    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:32:11.276957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.276934    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.318057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.316909    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:32:11.336652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.327121    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d0755f\""
Oct 31 16:32:11.336652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.328103    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d0755f-dockercfg-th5lb\""
Oct 31 16:32:11.363642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.357359    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnsm\" (UniqueName: \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.363642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.357567    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.534700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.534491    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnsm\" (UniqueName: \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.534700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.534669    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.567667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.535736    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.817233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.817199    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:32:11.844861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.844822    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnsm\" (UniqueName: \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\") pod \"payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.913644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.905017    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:11.927641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.927036    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:32:11.927641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.927168    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:11.952841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:11.952811    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:32:12.011417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.008547    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7kl\" (UniqueName: \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.011417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.008685    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.111119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.111080    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7kl\" (UniqueName: \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.111273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.111159    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.112194 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.112166    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.167676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.167604    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7kl\" (UniqueName: \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\") pod \"payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.382709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.382599    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:12.424137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.424107    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:32:12.425340 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:12.425313    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4d932e_cdd9_4e04_ab7f_b849d8f55120.slice/crio-45f56e8aee0198dfe938bee0dae8ffb6b8c2195859e245001954b2e0f1944717 WatchSource:0}: Error finding container 45f56e8aee0198dfe938bee0dae8ffb6b8c2195859e245001954b2e0f1944717: Status 404 returned error can't find the container with id 45f56e8aee0198dfe938bee0dae8ffb6b8c2195859e245001954b2e0f1944717
Oct 31 16:32:12.785343 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:12.785299    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c419478_af60_4db4_865e_b644f4b9164f.slice/crio-d4274e5c4fe36fe60d18ee370f5096dcf5fe1b2e5c7b2dfb1d25621cd6390a48 WatchSource:0}: Error finding container d4274e5c4fe36fe60d18ee370f5096dcf5fe1b2e5c7b2dfb1d25621cd6390a48: Status 404 returned error can't find the container with id d4274e5c4fe36fe60d18ee370f5096dcf5fe1b2e5c7b2dfb1d25621cd6390a48
Oct 31 16:32:12.786463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.786436    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:32:12.809565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.809526    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" event={"ID":"4e4d932e-cdd9-4e04-ab7f-b849d8f55120","Type":"ContainerStarted","Data":"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"}
Oct 31 16:32:12.809863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.809826    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:12.809863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.809862    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" event={"ID":"4e4d932e-cdd9-4e04-ab7f-b849d8f55120","Type":"ContainerStarted","Data":"45f56e8aee0198dfe938bee0dae8ffb6b8c2195859e245001954b2e0f1944717"}
Oct 31 16:32:12.811309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.811260    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" event={"ID":"4c419478-af60-4db4-865e-b644f4b9164f","Type":"ContainerStarted","Data":"d4274e5c4fe36fe60d18ee370f5096dcf5fe1b2e5c7b2dfb1d25621cd6390a48"}
Oct 31 16:32:12.920426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:12.920368    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" podStartSLOduration=1.9203517909999999 podStartE2EDuration="1.920351791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:12.915596742 +0000 UTC m=+1777.977171927" watchObservedRunningTime="2026-10-31 16:32:12.920351791 +0000 UTC m=+1777.981926978"
Oct 31 16:32:14.013390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.013347    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:32:14.019864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.019705    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:32:14.064552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.025709    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.064552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.046702    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" event={"ID":"4c419478-af60-4db4-865e-b644f4b9164f","Type":"ContainerStarted","Data":"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"}
Oct 31 16:32:14.064552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.046765    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:14.118639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.117226    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3dc1da\""
Oct 31 16:32:14.118639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.117912    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3dc1da-dockercfg-qrkv9\""
Oct 31 16:32:14.217989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.217950    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.218561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.218532    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgnd\" (UniqueName: \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.289255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.289155    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" podStartSLOduration=3.289136991 podStartE2EDuration="3.289136991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:14.283888278 +0000 UTC m=+1779.345463464" watchObservedRunningTime="2026-10-31 16:32:14.289136991 +0000 UTC m=+1779.350712178"
Oct 31 16:32:14.319824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.319787    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.320019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.320006    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgnd\" (UniqueName: \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.320535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.320509    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.372542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.372507    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgnd\" (UniqueName: \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\") pod \"payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") " pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.467762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.453369    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:14.906100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:14.906029    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:32:14.907521 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:14.907487    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d61af7_0e80_46f0_b412_742398967894.slice/crio-c7ab64e2a22c1c8cddd3c887c022515e29ddda5a31dffd41a05dc284b4f76444 WatchSource:0}: Error finding container c7ab64e2a22c1c8cddd3c887c022515e29ddda5a31dffd41a05dc284b4f76444: Status 404 returned error can't find the container with id c7ab64e2a22c1c8cddd3c887c022515e29ddda5a31dffd41a05dc284b4f76444
Oct 31 16:32:15.096133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.094220    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" event={"ID":"c9d61af7-0e80-46f0-b412-742398967894","Type":"ContainerStarted","Data":"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"}
Oct 31 16:32:15.096133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.094302    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" event={"ID":"c9d61af7-0e80-46f0-b412-742398967894","Type":"ContainerStarted","Data":"c7ab64e2a22c1c8cddd3c887c022515e29ddda5a31dffd41a05dc284b4f76444"}
Oct 31 16:32:15.096133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.094453    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:15.201965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.201864    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" podStartSLOduration=2.201850286 podStartE2EDuration="2.201850286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:15.199663013 +0000 UTC m=+1780.261238204" watchObservedRunningTime="2026-10-31 16:32:15.201850286 +0000 UTC m=+1780.263425472"
Oct 31 16:32:15.356643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.356003    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:32:15.368645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:15.363745    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:32:19.199267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.199228    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:32:19.204224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.204196    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221428    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221489    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221547    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221593    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmml\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221680    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221720    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221858    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221891    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.221942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.221932    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.291889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.291851    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:32:19.404752 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.362376    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.431789    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.431875    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmml\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.431918    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.431970    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.432143    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.432192    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.432241    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.432373    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.432858    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.433060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.375311    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.434132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.433720    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.434132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.433980    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.434285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.434207    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.438845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.438814    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.448335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.448283    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.487389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.487308    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmml\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.487681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.487660    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.520600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.520562    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:19.966117 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:19.966071    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134591d3_3561_4c21_96f7_f2f00b98f9ba.slice/crio-52a01fb39b7c731ba867fe66c041dd0c5e5d4a32909d7ee63ae91e582ea1a716 WatchSource:0}: Error finding container 52a01fb39b7c731ba867fe66c041dd0c5e5d4a32909d7ee63ae91e582ea1a716: Status 404 returned error can't find the container with id 52a01fb39b7c731ba867fe66c041dd0c5e5d4a32909d7ee63ae91e582ea1a716
Oct 31 16:32:19.966590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.966559    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:32:19.969291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.969251    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:19.969406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.969326    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:19.969406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:19.969361    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.137982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.137945    4417 generic.go:356] "Generic (PLEG): container finished" podID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerID="ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea" exitCode=0
Oct 31 16:32:20.138290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.137991    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerDied","Data":"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"}
Oct 31 16:32:20.138671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.138636    4417 scope.go:122] "RemoveContainer" containerID="ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"
Oct 31 16:32:20.140751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.140718    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" event={"ID":"134591d3-3561-4c21-96f7-f2f00b98f9ba","Type":"ContainerStarted","Data":"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"}
Oct 31 16:32:20.140851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.140779    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" event={"ID":"134591d3-3561-4c21-96f7-f2f00b98f9ba","Type":"ContainerStarted","Data":"52a01fb39b7c731ba867fe66c041dd0c5e5d4a32909d7ee63ae91e582ea1a716"}
Oct 31 16:32:20.141273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.141239    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.141351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.141334    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.141404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.141385    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:32:20.395132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.395063    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podStartSLOduration=1.395043472 podStartE2EDuration="1.395043472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:20.368222554 +0000 UTC m=+1785.429797766" watchObservedRunningTime="2026-10-31 16:32:20.395043472 +0000 UTC m=+1785.456618664"
Oct 31 16:32:20.521567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:20.521529    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:21.171640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.170671    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerStarted","Data":"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"}
Oct 31 16:32:21.523209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.523120    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:21.523209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.523193    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:21.968643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.968364    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:32:21.968643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.968633    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:21.968920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:21.968688    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:22.605641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:22.586551    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:22.605641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:22.586646    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:22.795732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:22.786377    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:32:23.025610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.003646    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:32:23.025610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.003781    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.033427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.032793    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:23.033427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.032938    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:23.099410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.098708    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bdc828\""
Oct 31 16:32:23.169777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.100150    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bdc828-dockercfg-xc7gm\""
Oct 31 16:32:23.205183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.205131    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.205475 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.205445    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9gx\" (UniqueName: \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.307317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.307250    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.307590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.307570    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9gx\" (UniqueName: \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.308598 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.308547    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.410921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.410894    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9gx\" (UniqueName: \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\") pod \"payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.481994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.481910    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:23.522555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.522519    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:23.522689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.522575    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:23.927791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.927762    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:32:23.929870 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:32:23.929840    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3119ac_4414_4f7f_94db_8478555dc8d5.slice/crio-1048628fbfb2dc1f062d5e4a68c87a513e0c97ed87dfb8b91a3a6142a591d0bd WatchSource:0}: Error finding container 1048628fbfb2dc1f062d5e4a68c87a513e0c97ed87dfb8b91a3a6142a591d0bd: Status 404 returned error can't find the container with id 1048628fbfb2dc1f062d5e4a68c87a513e0c97ed87dfb8b91a3a6142a591d0bd
Oct 31 16:32:23.973665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.968955    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:23.973665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:23.969015    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:24.097025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.096515    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:32:24.324249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.324208    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" event={"ID":"2c3119ac-4414-4f7f-94db-8478555dc8d5","Type":"ContainerStarted","Data":"7aea6d34e456a5112a7977dc2a565b61d6ea2621a2933c3d03fab191c8ebb996"}
Oct 31 16:32:24.324421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.324275    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" event={"ID":"2c3119ac-4414-4f7f-94db-8478555dc8d5","Type":"ContainerStarted","Data":"1048628fbfb2dc1f062d5e4a68c87a513e0c97ed87dfb8b91a3a6142a591d0bd"}
Oct 31 16:32:24.324815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.324606    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:32:24.432445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.432395    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" podStartSLOduration=2.432381568 podStartE2EDuration="2.432381568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:24.430481415 +0000 UTC m=+1789.492056601" watchObservedRunningTime="2026-10-31 16:32:24.432381568 +0000 UTC m=+1789.493956798"
Oct 31 16:32:24.521159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.521075    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:24.521392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.521373    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:24.964642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.963050    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:24.964642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:24.963118    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:25.105414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.104335    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:32:25.521243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.521211    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:25.521418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.521277    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:25.786933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.786826    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:32:25.961641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.961009    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:25.961641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:25.961077    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:26.108137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:26.108108    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:32:26.521069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:26.520967    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 16:32:26.521069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:26.521027    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 16:32:26.968806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:26.968753    4417 patch_prober.go:29] interesting pod/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 16:32:26.969108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:26.969081    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 16:32:27.526633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:27.526589    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:27.527105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:27.527086    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:27.527772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:27.527750    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:32:27.965219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:27.965184    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:32:27.966480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:27.966452    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:32:32.946902 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:32:32.946863    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:32:35.416139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:32:35.416108    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:33:02.947453 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:33:02.947412    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:33:04.081678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.081641    4417 generic.go:356] "Generic (PLEG): container finished" podID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerID="2124dba75bb820e75ee3c83236da1ae89debddc4f0515779aebc8f4d11baef0c" exitCode=0
Oct 31 16:33:04.082134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.081692    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerDied","Data":"2124dba75bb820e75ee3c83236da1ae89debddc4f0515779aebc8f4d11baef0c"}
Oct 31 16:33:04.082266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.082247    4417 scope.go:122] "RemoveContainer" containerID="2124dba75bb820e75ee3c83236da1ae89debddc4f0515779aebc8f4d11baef0c"
Oct 31 16:33:04.084255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.084226    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:04.084341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.084300    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:04.084341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:04.084332    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:33:05.091922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:05.091884    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerStarted","Data":"812026b273c375c3301e032bf35c1ef4fa6cb6d0134fa73b2e84e9decfe8ea8d"}
Oct 31 16:33:05.799714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:05.799673    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:33:05.801072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:05.801042    4417 patch_prober.go:29] interesting pod/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 16:33:05.801218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:05.801090    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 16:33:06.799903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:06.799867    4417 patch_prober.go:29] interesting pod/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 16:33:06.800409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:06.799937    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 16:33:07.800118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:07.800077    4417 patch_prober.go:29] interesting pod/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 16:33:07.800538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:07.800142    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 16:33:07.874528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:07.874479    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:33:08.799561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:08.799525    4417 patch_prober.go:29] interesting pod/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 16:33:08.799778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:08.799596    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 16:33:09.799245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:09.799203    4417 patch_prober.go:29] interesting pod/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 31 16:33:09.799659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:09.799280    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 31 16:33:10.803898 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:10.803863    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:33:10.805228 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:10.805206    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:33:32.950897 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:33:32.950848    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:33:48.409658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:48.409607    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:33:48.511567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:48.511522    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:33:51.020499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:51.020457    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:33:51.121921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:51.121886    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:33:53.410249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.410206    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" podUID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" containerName="payload-processing" containerID="cri-o://3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e" gracePeriod=25
Oct 31 16:33:53.512512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.512467    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" podUID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" containerName="payload-pre-processing" containerID="cri-o://6f91ac95c0ad72805700e332f909906639c6e09be688dd00a2c38d5a705e96c2" gracePeriod=25
Oct 31 16:33:53.714098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.714069    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:33:53.773066 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.773029    4417 generic.go:356] "Generic (PLEG): container finished" podID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" containerID="3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e" exitCode=0
Oct 31 16:33:53.773400 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.773075    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"
Oct 31 16:33:53.773920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.773097    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" event={"ID":"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b","Type":"ContainerDied","Data":"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"}
Oct 31 16:33:53.773920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.773744    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99" event={"ID":"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b","Type":"ContainerDied","Data":"bcb1cfb02e7953cb6d48cd02feb199d672e34d0b7aa64fe10abf22d2b836f034"}
Oct 31 16:33:53.773920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.773820    4417 scope.go:122] "RemoveContainer" containerID="3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"
Oct 31 16:33:53.776576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.776280    4417 generic.go:356] "Generic (PLEG): container finished" podID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" containerID="6f91ac95c0ad72805700e332f909906639c6e09be688dd00a2c38d5a705e96c2" exitCode=0
Oct 31 16:33:53.776576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.776480    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" event={"ID":"c1b2d24f-8d3a-4090-85ce-ce1815f100c3","Type":"ContainerDied","Data":"6f91ac95c0ad72805700e332f909906639c6e09be688dd00a2c38d5a705e96c2"}
Oct 31 16:33:53.790506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.790375    4417 scope.go:122] "RemoveContainer" containerID="3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"
Oct 31 16:33:53.790798 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:33:53.790767    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e\": container with ID starting with 3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e not found: ID does not exist" containerID="3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"
Oct 31 16:33:53.790895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.790812    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e"} err="failed to get container status \"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e\": rpc error: code = NotFound desc = could not find container \"3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e\": container with ID starting with 3c87a9d47a2857c6dd108b71f5397431cf543dfbb9c160a0430f1eee0cfcab0e not found: ID does not exist"
Oct 31 16:33:53.806609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.806581    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\" (UniqueName: \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\") pod \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") "
Oct 31 16:33:53.806892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.806877    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\") pod \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\" (UID: \"0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b\") "
Oct 31 16:33:53.807282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.807255    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume" pod "0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" (UID: "0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:33:53.807479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.807454    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:53.808951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.808916    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7" pod "0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" (UID: "0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b"). InnerVolumeSpecName "kube-api-access-fbqc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:33:53.842670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.838985    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:33:53.908083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.908047    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\" (UniqueName: \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\") pod \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") "
Oct 31 16:33:53.908293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.908226    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\") pod \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\" (UID: \"c1b2d24f-8d3a-4090-85ce-ce1815f100c3\") "
Oct 31 16:33:53.908590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.908563    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume" pod "c1b2d24f-8d3a-4090-85ce-ce1815f100c3" (UID: "c1b2d24f-8d3a-4090-85ce-ce1815f100c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:33:53.908838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.908821    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:53.908891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.908843    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqc7\" (UniqueName: \"kubernetes.io/projected/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b-kube-api-access-fbqc7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:53.910175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:53.910156    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh" pod "c1b2d24f-8d3a-4090-85ce-ce1815f100c3" (UID: "c1b2d24f-8d3a-4090-85ce-ce1815f100c3"). InnerVolumeSpecName "kube-api-access-gsjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:33:54.010876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.010790    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjnh\" (UniqueName: \"kubernetes.io/projected/c1b2d24f-8d3a-4090-85ce-ce1815f100c3-kube-api-access-gsjnh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:33:54.195084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.195043    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:33:54.200571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.200541    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e977aa-bbc97bd5f-swn99"]
Oct 31 16:33:54.787255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.787169    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"
Oct 31 16:33:54.787255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.787185    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8" event={"ID":"c1b2d24f-8d3a-4090-85ce-ce1815f100c3","Type":"ContainerDied","Data":"d1ac115cb0d4e590b30f667ae1a4728d635d3c72e7f35ebe720bb0173d4d611e"}
Oct 31 16:33:54.787870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.787274    4417 scope.go:122] "RemoveContainer" containerID="6f91ac95c0ad72805700e332f909906639c6e09be688dd00a2c38d5a705e96c2"
Oct 31 16:33:54.940681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.940645    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:33:54.987938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:54.987900    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e977aa-769db8bb9f-9bnh8"]
Oct 31 16:33:55.053057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:55.053026    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" path="/var/lib/kubelet/pods/0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b/volumes"
Oct 31 16:33:55.053499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:55.053486    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" path="/var/lib/kubelet/pods/c1b2d24f-8d3a-4090-85ce-ce1815f100c3/volumes"
Oct 31 16:33:56.021180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:56.021141    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing" containerID="cri-o://3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486" gracePeriod=25
Oct 31 16:33:56.123213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:56.123167    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing" containerID="cri-o://f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b" gracePeriod=25
Oct 31 16:33:59.535867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.535827    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:33:59.536929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.536902    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" containerName="payload-pre-processing"
Oct 31 16:33:59.536929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.536928    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" containerName="payload-pre-processing"
Oct 31 16:33:59.537127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.536958    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" containerName="payload-processing"
Oct 31 16:33:59.537127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.536966    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" containerName="payload-processing"
Oct 31 16:33:59.537127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.537120    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b2d24f-8d3a-4090-85ce-ce1815f100c3" containerName="payload-pre-processing"
Oct 31 16:33:59.537280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.537136    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5c9200-84b0-4b20-bff8-7a54dc6fcf3b" containerName="payload-processing"
Oct 31 16:33:59.540866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.540844    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.624266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.624224    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.624527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.624494    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprrl\" (UniqueName: \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.624664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.624641    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.624829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.624805    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.636792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.636745    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bdc828-serving-cert\""
Oct 31 16:33:59.643106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.643071    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:33:59.726940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.726906    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprrl\" (UniqueName: \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.727162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.726960    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.727162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.727033    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.727162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.727158    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.727325 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:33:59.727167    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-serving-cert: secret "maas-api-e2e-api-a-bdc828-serving-cert" not found
Oct 31 16:33:59.727325 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:33:59.727230    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls podName:439a64af-1645-4690-bc3b-1b5cf856c432 nodeName:}" failed. No retries permitted until 2026-10-31 16:34:00.227209602 +0000 UTC m=+1885.288784766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls") pod "maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" (UID: "439a64af-1645-4690-bc3b-1b5cf856c432") : secret "maas-api-e2e-api-a-bdc828-serving-cert" not found
Oct 31 16:33:59.727894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.727872    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.729586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.729543    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:33:59.782643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:33:59.782193    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprrl\" (UniqueName: \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:00.082259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.082214    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:34:00.082436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.082264    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:34:00.258452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.258409    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:00.261732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.261698    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") pod \"maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:00.455833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.455740    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:00.837118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.837082    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:34:00.839547 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:34:00.839513    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439a64af_1645_4690_bc3b_1b5cf856c432.slice/crio-3bbf4992005505450bb6d396b14a8b1776a3fac9d11774f535336e08d59bde5a WatchSource:0}: Error finding container 3bbf4992005505450bb6d396b14a8b1776a3fac9d11774f535336e08d59bde5a: Status 404 returned error can't find the container with id 3bbf4992005505450bb6d396b14a8b1776a3fac9d11774f535336e08d59bde5a
Oct 31 16:34:00.879691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:00.879651    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" event={"ID":"439a64af-1645-4690-bc3b-1b5cf856c432","Type":"ContainerStarted","Data":"3bbf4992005505450bb6d396b14a8b1776a3fac9d11774f535336e08d59bde5a"}
Oct 31 16:34:01.892322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:01.892284    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" event={"ID":"439a64af-1645-4690-bc3b-1b5cf856c432","Type":"ContainerStarted","Data":"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"}
Oct 31 16:34:01.892746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:01.892367    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:02.028479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:02.028421    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" podStartSLOduration=2.088476222 podStartE2EDuration="3.028407561s" totalImagesPullingTime="939.931339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:02.026547008 +0000 UTC m=+1887.088122198" watchObservedRunningTime="2026-10-31 16:34:02.028407561 +0000 UTC m=+1887.089982747"
Oct 31 16:34:02.948878 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:02.948836    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:34:07.907722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:07.907692    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:34:10.081990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:10.081942    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:34:10.082492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:10.082019    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:34:16.333168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.333136    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:34:16.432300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.432273    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:34:16.441394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.441367    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\") pod \"8ae53aae-528c-451c-a9cd-d0f41955daeb\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") "
Oct 31 16:34:16.441583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.441569    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\" (UniqueName: \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\") pod \"8ae53aae-528c-451c-a9cd-d0f41955daeb\" (UID: \"8ae53aae-528c-451c-a9cd-d0f41955daeb\") "
Oct 31 16:34:16.441867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.441841    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume" pod "8ae53aae-528c-451c-a9cd-d0f41955daeb" (UID: "8ae53aae-528c-451c-a9cd-d0f41955daeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:16.442307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.442279    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae53aae-528c-451c-a9cd-d0f41955daeb-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:16.443830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.443803    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b" pod "8ae53aae-528c-451c-a9cd-d0f41955daeb" (UID: "8ae53aae-528c-451c-a9cd-d0f41955daeb"). InnerVolumeSpecName "kube-api-access-vtd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:16.542894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.542860    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\" (UniqueName: \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\") pod \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") "
Oct 31 16:34:16.543099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.543003    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\") pod \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\" (UID: \"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e\") "
Oct 31 16:34:16.543416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.543389    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume" pod "2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" (UID: "2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:16.543832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.543802    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtd9b\" (UniqueName: \"kubernetes.io/projected/8ae53aae-528c-451c-a9cd-d0f41955daeb-kube-api-access-vtd9b\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:16.543832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.543827    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:16.545486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.545422    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh" pod "2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" (UID: "2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e"). InnerVolumeSpecName "kube-api-access-687bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:16.646055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:16.646006    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687bh\" (UniqueName: \"kubernetes.io/projected/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e-kube-api-access-687bh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:17.046113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.046076    4417 generic.go:356] "Generic (PLEG): container finished" podID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerID="f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b" exitCode=0
Oct 31 16:34:17.046276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.046265    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"
Oct 31 16:34:17.048005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.047975    4417 generic.go:356] "Generic (PLEG): container finished" podID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerID="3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486" exitCode=0
Oct 31 16:34:17.048151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.048078    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"
Oct 31 16:34:17.057409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.057247    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" event={"ID":"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e","Type":"ContainerDied","Data":"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"}
Oct 31 16:34:17.057409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.057288    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm" event={"ID":"2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e","Type":"ContainerDied","Data":"77c21f68c6fe598e144e3eae78d71016d31952048afd5781063bd151b45ad69d"}
Oct 31 16:34:17.057409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.057310    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" event={"ID":"8ae53aae-528c-451c-a9cd-d0f41955daeb","Type":"ContainerDied","Data":"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"}
Oct 31 16:34:17.057409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.057331    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm" event={"ID":"8ae53aae-528c-451c-a9cd-d0f41955daeb","Type":"ContainerDied","Data":"15ecaab91763aa5072d9b2c1521b0f1699fe0566033b296cda10caaa73edc2ee"}
Oct 31 16:34:17.057409 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.057363    4417 scope.go:122] "RemoveContainer" containerID="f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"
Oct 31 16:34:17.072085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.072054    4417 scope.go:122] "RemoveContainer" containerID="f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"
Oct 31 16:34:17.072487 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:17.072456    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b\": container with ID starting with f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b not found: ID does not exist" containerID="f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"
Oct 31 16:34:17.072536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.072497    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b"} err="failed to get container status \"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b\": rpc error: code = NotFound desc = could not find container \"f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b\": container with ID starting with f76b3cf2e6f6caed81487d4bf4676fe0d404bc240b79f965a927eda8ec41282b not found: ID does not exist"
Oct 31 16:34:17.072536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.072516    4417 scope.go:122] "RemoveContainer" containerID="3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"
Oct 31 16:34:17.084352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.084330    4417 scope.go:122] "RemoveContainer" containerID="3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"
Oct 31 16:34:17.084638 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:17.084605    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486\": container with ID starting with 3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486 not found: ID does not exist" containerID="3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"
Oct 31 16:34:17.084704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.084649    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486"} err="failed to get container status \"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486\": rpc error: code = NotFound desc = could not find container \"3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486\": container with ID starting with 3d57a4e7be318aecb6104d2fa4b541fb7fb673a683b184b5f6d2183af86d7486 not found: ID does not exist"
Oct 31 16:34:17.204868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.204832    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:34:17.250014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.249974    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-6zhpm"]
Oct 31 16:34:17.350323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.350231    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:34:17.354315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:17.354287    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-4cxlm"]
Oct 31 16:34:19.052629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:19.052580    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" path="/var/lib/kubelet/pods/2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e/volumes"
Oct 31 16:34:19.053411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:19.053388    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" path="/var/lib/kubelet/pods/8ae53aae-528c-451c-a9cd-d0f41955daeb/volumes"
Oct 31 16:34:32.946444 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:32.946406    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:34:33.573518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.573458    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:34:33.574415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574394    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing"
Oct 31 16:34:33.574415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574415    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing"
Oct 31 16:34:33.574551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574454    4417 cpu_manager.go:414] "containerMap: removing container" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing"
Oct 31 16:34:33.574551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574460    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing"
Oct 31 16:34:33.574691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574575    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3827b8-a2ce-42af-aa8a-b4b5d9e3909e" containerName="payload-pre-processing"
Oct 31 16:34:33.574691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.574590    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae53aae-528c-451c-a9cd-d0f41955daeb" containerName="payload-processing"
Oct 31 16:34:33.579215 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.579191    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.627817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.627780    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a2ae37-serving-cert\""
Oct 31 16:34:33.632135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.632101    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:34:33.649464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.649426    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.649678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.649631    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4298x\" (UniqueName: \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.649761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.649735    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.649880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.649859    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.751030 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.750994    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.751220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.751118    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4298x\" (UniqueName: \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.751220 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:33.751174    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-serving-cert: secret "maas-api-e2e-inf-a-a2ae37-serving-cert" not found
Oct 31 16:34:33.751341 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:33.751262    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls podName:ac029bba-1128-4098-8c80-268574190a1c nodeName:}" failed. No retries permitted until 2026-10-31 16:34:34.25123989 +0000 UTC m=+1919.312815053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls") pod "maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" (UID: "ac029bba-1128-4098-8c80-268574190a1c") : secret "maas-api-e2e-inf-a-a2ae37-serving-cert" not found
Oct 31 16:34:33.751341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.751178    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.751461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.751381    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.752081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.752055    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.753669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.753650    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:33.806395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:33.806355    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4298x\" (UniqueName: \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:34.262560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:34.262522    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:34.265045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:34.265022    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:34.501604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:34.501564    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:34.877847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:34.877815    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:34:34.878658 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:34:34.878629    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac029bba_1128_4098_8c80_268574190a1c.slice/crio-190304d229203c7f8e3abbc34326ca1dadcbe2d07ba7725e48754b9f70ab0656 WatchSource:0}: Error finding container 190304d229203c7f8e3abbc34326ca1dadcbe2d07ba7725e48754b9f70ab0656: Status 404 returned error can't find the container with id 190304d229203c7f8e3abbc34326ca1dadcbe2d07ba7725e48754b9f70ab0656
Oct 31 16:34:35.258401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:35.258302    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" event={"ID":"ac029bba-1128-4098-8c80-268574190a1c","Type":"ContainerStarted","Data":"190304d229203c7f8e3abbc34326ca1dadcbe2d07ba7725e48754b9f70ab0656"}
Oct 31 16:34:36.267349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:36.267250    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" event={"ID":"ac029bba-1128-4098-8c80-268574190a1c","Type":"ContainerStarted","Data":"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"}
Oct 31 16:34:36.267818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:36.267498    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:36.375524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:36.375471    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" podStartSLOduration=2.457957421 podStartE2EDuration="3.375457361s" totalImagesPullingTime="917.49994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:36.373124264 +0000 UTC m=+1921.434699451" watchObservedRunningTime="2026-10-31 16:34:36.375457361 +0000 UTC m=+1921.437032547"
Oct 31 16:34:41.453805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.453763    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:34:41.454239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.454009    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" containerID="cri-o://7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3" gracePeriod=30
Oct 31 16:34:41.497446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.497415    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:34:41.497675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.497652    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy" containerID="cri-o://969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047" gracePeriod=30
Oct 31 16:34:41.602323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.602282    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:34:41.602659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:41.602579    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" containerID="cri-o://63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a" gracePeriod=30
Oct 31 16:34:42.280830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.280798    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:34:42.294323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.294290    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:34:42.294552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.294518    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery" containerID="cri-o://565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3" gracePeriod=30
Oct 31 16:34:42.634542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.634512    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:34:42.636784 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.636762    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.636944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.636832    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.636944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.636892    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.636944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.636933    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.637135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.637002    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.637135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.637100    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.637263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.637241    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\") pod \"86d71e33-c1dc-4fea-ad9d-344f28388051\" (UID: \"86d71e33-c1dc-4fea-ad9d-344f28388051\") "
Oct 31 16:34:42.637263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.637254    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:42.637847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.637817    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-ca-configmap\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.639546    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.639873    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.639900    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.639911    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.639949    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:42.640657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.640118    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb" pod "86d71e33-c1dc-4fea-ad9d-344f28388051" (UID: "86d71e33-c1dc-4fea-ad9d-344f28388051"). InnerVolumeSpecName "kube-api-access-s57mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:42.739358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739320    4417 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d71e33-c1dc-4fea-ad9d-344f28388051-local-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.739358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739349    4417 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-cacerts\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.739358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739359    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.739358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739368    4417 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-kubeconfig\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.739765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739379    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86d71e33-c1dc-4fea-ad9d-344f28388051-istio-csr-dns-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:42.739765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:42.739388    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57mb\" (UniqueName: \"kubernetes.io/projected/86d71e33-c1dc-4fea-ad9d-344f28388051-kube-api-access-s57mb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:43.328772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.328677    4417 generic.go:356] "Generic (PLEG): container finished" podID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerID="565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3" exitCode=0
Oct 31 16:34:43.328772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.328742    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"
Oct 31 16:34:43.328772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.328750    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" event={"ID":"86d71e33-c1dc-4fea-ad9d-344f28388051","Type":"ContainerDied","Data":"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"}
Oct 31 16:34:43.329045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.328799    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d" event={"ID":"86d71e33-c1dc-4fea-ad9d-344f28388051","Type":"ContainerDied","Data":"d101fc2a60ad4ad9601411f449822e48558eff0f2b26ef76d0df93f7da827f60"}
Oct 31 16:34:43.329045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.328820    4417 scope.go:122] "RemoveContainer" containerID="565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"
Oct 31 16:34:43.339905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.339883    4417 scope.go:122] "RemoveContainer" containerID="565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"
Oct 31 16:34:43.340182 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:43.340158    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3\": container with ID starting with 565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3 not found: ID does not exist" containerID="565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"
Oct 31 16:34:43.340279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.340189    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3"} err="failed to get container status \"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3\": rpc error: code = NotFound desc = could not find container \"565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3\": container with ID starting with 565585cf01cdf56b332b93f32869f7b244b723934c9c1b546ee709dd7624bfb3 not found: ID does not exist"
Oct 31 16:34:43.383532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.383505    4417 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:34:43.383686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.383558    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:34:43.484264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.484233    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:34:43.489349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.489321    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzm9d"]
Oct 31 16:34:43.752173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.752080    4417 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:34:43.753163 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:43.752730    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:34:44.317177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.317143    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:34:44.318281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.318263    4417 cpu_manager.go:414] "containerMap: removing container" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery"
Oct 31 16:34:44.318377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.318282    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery"
Oct 31 16:34:44.318483 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.318471    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" containerName="discovery"
Oct 31 16:34:44.324098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.324078    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361045    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361119    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361164    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361202    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361254    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361369    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361424    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361480    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.361650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.361629    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6ps\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.372479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.372457    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdc828-openshift-default-dockercfg-gq77f\""
Oct 31 16:34:44.419918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.419893    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:34:44.463092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463049    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6ps\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463181    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463252    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463293    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463328    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463384    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463764 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463739    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463796    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463895    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.463954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463938    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.464064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.463984    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.464328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.464253    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.464647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.464604    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.464853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.464829    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.466185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.466163    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.466748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.466729    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.515573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.515543    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6ps\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.516503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.516479    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:44.642329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:44.642249    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:45.104493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.104451    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:45.104943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.104554    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:45.104943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.104637    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:34:45.113211 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.113181    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d71e33-c1dc-4fea-ad9d-344f28388051" path="/var/lib/kubelet/pods/86d71e33-c1dc-4fea-ad9d-344f28388051/volumes"
Oct 31 16:34:45.113711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.113689    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:34:45.350037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.349948    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" event={"ID":"52379bc0-1166-4cdb-840c-7d44348dee1a","Type":"ContainerStarted","Data":"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"}
Oct 31 16:34:45.350037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.350026    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" event={"ID":"52379bc0-1166-4cdb-840c-7d44348dee1a","Type":"ContainerStarted","Data":"6df62a53434b4674280da33a07612f70881962b01a56dc1938d6375e8e35511a"}
Oct 31 16:34:45.459509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.459457    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podStartSLOduration=1.459442069 podStartE2EDuration="1.459442069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:45.456895924 +0000 UTC m=+1930.518471111" watchObservedRunningTime="2026-10-31 16:34:45.459442069 +0000 UTC m=+1930.521017254"
Oct 31 16:34:45.642843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:45.642757    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:46.667645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:46.662788    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:46.667645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:46.662865    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:46.906940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:46.906917    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:34:46.912144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:46.912009    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:34:47.026833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026805    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.026971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026845    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.026971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026882    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.026971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026907    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.026971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026933    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.026969    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027053    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027104    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027150    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.027422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.027422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027238    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.027422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027306    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027438    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.027587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027499    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027522    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\") pod \"a8191271-030d-4170-b05e-ac9c285d8f4d\" (UID: \"a8191271-030d-4170-b05e-ac9c285d8f4d\") "
Oct 31 16:34:47.027587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027552    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027124    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.027863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027186    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.027863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027605    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.027863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027651    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\") pod \"dfc1ac91-b961-4fd1-b498-3c774e480a17\" (UID: \"dfc1ac91-b961-4fd1-b498-3c774e480a17\") "
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027287    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028505    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028566    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028584    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028593    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027514    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027781    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.027860    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028074    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028638    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.028669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.028204    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.030652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.030584    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.030760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.030688    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.031297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.031271    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:47.031297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.031291    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "kube-api-access-5k2sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.031446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.031278    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "kube-api-access-4wsmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.031596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.031575    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:47.031700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.031577    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy" pod "a8191271-030d-4170-b05e-ac9c285d8f4d" (UID: "a8191271-030d-4170-b05e-ac9c285d8f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.032086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.032067    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token" pod "dfc1ac91-b961-4fd1-b498-3c774e480a17" (UID: "dfc1ac91-b961-4fd1-b498-3c774e480a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.041722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.040864    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:34:47.129032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.128990    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129143    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129196    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129244    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129278    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.129371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129309    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129356    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129390    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129541    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.129957 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129589    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\") pod \"1d1d880b-ff80-4de9-b6ec-eef487720152\" (UID: \"1d1d880b-ff80-4de9-b6ec-eef487720152\") "
Oct 31 16:34:47.130116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.129996    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:34:47.130172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130109    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.130482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130454    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130487    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130495    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130521    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130537    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130535    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130551    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k2sw\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-kube-api-access-5k2sw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.130574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130567    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130581    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130595    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130609    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130645    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8191271-030d-4170-b05e-ac9c285d8f4d-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130659    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130674    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130686    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1ac91-b961-4fd1-b498-3c774e480a17-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130699    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8191271-030d-4170-b05e-ac9c285d8f4d-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130711    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1d880b-ff80-4de9-b6ec-eef487720152-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130724    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1ac91-b961-4fd1-b498-3c774e480a17-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130738    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wsmd\" (UniqueName: \"kubernetes.io/projected/a8191271-030d-4170-b05e-ac9c285d8f4d-kube-api-access-4wsmd\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.131015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.130751    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1ac91-b961-4fd1-b498-3c774e480a17-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.132469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.132432    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "kube-api-access-zb57r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.132671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.132627    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:34:47.133084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.133062    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:34:47.154704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.143082    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy" pod "1d1d880b-ff80-4de9-b6ec-eef487720152" (UID: "1d1d880b-ff80-4de9-b6ec-eef487720152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:34:47.232963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.232927    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.232963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.232963    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.233283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.232979    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.233283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.232993    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb57r\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-kube-api-access-zb57r\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.233283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.233006    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1d880b-ff80-4de9-b6ec-eef487720152-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.233283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.233018    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1d880b-ff80-4de9-b6ec-eef487720152-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:34:47.401285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.401250    4417 generic.go:356] "Generic (PLEG): container finished" podID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerID="969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047" exitCode=0
Oct 31 16:34:47.401493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.401319    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" event={"ID":"a8191271-030d-4170-b05e-ac9c285d8f4d","Type":"ContainerDied","Data":"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"}
Oct 31 16:34:47.401493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.401375    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47" event={"ID":"a8191271-030d-4170-b05e-ac9c285d8f4d","Type":"ContainerDied","Data":"fc5de55d8049b775d82b2845a4dd2770dcd3d3ebd7fef8bf9c24d55aa61e89be"}
Oct 31 16:34:47.401493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.401397    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"
Oct 31 16:34:47.401493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.401404    4417 scope.go:122] "RemoveContainer" containerID="969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"
Oct 31 16:34:47.403704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.403675    4417 generic.go:356] "Generic (PLEG): container finished" podID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerID="7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3" exitCode=0
Oct 31 16:34:47.403881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.403863    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"
Oct 31 16:34:47.404541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.404515    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" event={"ID":"dfc1ac91-b961-4fd1-b498-3c774e480a17","Type":"ContainerDied","Data":"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"}
Oct 31 16:34:47.404627 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.404573    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh" event={"ID":"dfc1ac91-b961-4fd1-b498-3c774e480a17","Type":"ContainerDied","Data":"4f65e3456405c8355485b59caf56c112a493e00ac91cfdd8d687eee4add8c022"}
Oct 31 16:34:47.406772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.406750    4417 generic.go:356] "Generic (PLEG): container finished" podID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerID="63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a" exitCode=0
Oct 31 16:34:47.410663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.407048    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" event={"ID":"1d1d880b-ff80-4de9-b6ec-eef487720152","Type":"ContainerDied","Data":"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"}
Oct 31 16:34:47.410663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.407104    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc" event={"ID":"1d1d880b-ff80-4de9-b6ec-eef487720152","Type":"ContainerDied","Data":"01a380fe124ce5a9fb96c18cb06b6afe4c48251e777112110d49d8438e7fb8ba"}
Oct 31 16:34:47.410663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.407172    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"
Oct 31 16:34:47.419065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.419046    4417 scope.go:122] "RemoveContainer" containerID="969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"
Oct 31 16:34:47.419355 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:47.419337    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047\": container with ID starting with 969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047 not found: ID does not exist" containerID="969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"
Oct 31 16:34:47.419420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.419363    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047"} err="failed to get container status \"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047\": rpc error: code = NotFound desc = could not find container \"969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047\": container with ID starting with 969032078bfaf1777fc1e4919d69040fd9b3baba5967214c8cb1f0827cfa0047 not found: ID does not exist"
Oct 31 16:34:47.419420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.419380    4417 scope.go:122] "RemoveContainer" containerID="7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"
Oct 31 16:34:47.429060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.429046    4417 scope.go:122] "RemoveContainer" containerID="7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"
Oct 31 16:34:47.429464 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:47.429433    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3\": container with ID starting with 7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3 not found: ID does not exist" containerID="7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"
Oct 31 16:34:47.429531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.429475    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3"} err="failed to get container status \"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3\": rpc error: code = NotFound desc = could not find container \"7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3\": container with ID starting with 7a1356b4f2df7cf62d9d888a3e34664c90697d8fb8d96a493e7aab4a0bc530c3 not found: ID does not exist"
Oct 31 16:34:47.429531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.429500    4417 scope.go:122] "RemoveContainer" containerID="63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"
Oct 31 16:34:47.491388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.490979    4417 scope.go:122] "RemoveContainer" containerID="63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"
Oct 31 16:34:47.491388 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:34:47.491355    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a\": container with ID starting with 63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a not found: ID does not exist" containerID="63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"
Oct 31 16:34:47.492009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.491381    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a"} err="failed to get container status \"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a\": rpc error: code = NotFound desc = could not find container \"63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a\": container with ID starting with 63cf2f077fdbd6b808384bc6b40adf0f0af8d0dc09c880b77f31f02f099f784a not found: ID does not exist"
Oct 31 16:34:47.574424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.574388    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:34:47.578937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.578910    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-z7gfh"]
Oct 31 16:34:47.654693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.645204    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:47.654693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.645260    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:47.673772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.673729    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:34:47.676905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.676833    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-65jtc"]
Oct 31 16:34:47.778910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.778596    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:34:47.782760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:47.782671    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-nbw47"]
Oct 31 16:34:48.687637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:48.676405    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:48.687637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:48.676470    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:49.080029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:49.079992    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" path="/var/lib/kubelet/pods/1d1d880b-ff80-4de9-b6ec-eef487720152/volumes"
Oct 31 16:34:49.080650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:49.080631    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" path="/var/lib/kubelet/pods/a8191271-030d-4170-b05e-ac9c285d8f4d/volumes"
Oct 31 16:34:49.081159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:49.081144    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" path="/var/lib/kubelet/pods/dfc1ac91-b961-4fd1-b498-3c774e480a17/volumes"
Oct 31 16:34:49.655648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:49.650061    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:49.655648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:49.650125    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:50.654641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:50.645005    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:50.654641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:50.645078    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:51.642586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:51.642546    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 31 16:34:51.642819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:51.642650    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 31 16:34:52.649164 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:52.649134    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:52.649681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:52.649648    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:34:52.650381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:34:52.650361    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:35:02.946325 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:02.946288    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:35:06.553890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:06.553852    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:35:06.653117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:06.653075    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:35:11.009577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.009537    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:35:11.107956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.107918    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:35:11.557997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.557944    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" podUID="93f57064-04ed-4fb1-98be-d49e9911384c" containerName="payload-processing" containerID="cri-o://215e0ab0b759ddead6736161f80f497699780e0f2c92c50edb1d938eb025f95c" gracePeriod=25
Oct 31 16:35:11.653836 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.653796    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" podUID="b4bbbab1-405f-4534-bce0-76ab238a4569" containerName="payload-pre-processing" containerID="cri-o://42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd" gracePeriod=25
Oct 31 16:35:11.745390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.745353    4417 generic.go:356] "Generic (PLEG): container finished" podID="93f57064-04ed-4fb1-98be-d49e9911384c" containerID="215e0ab0b759ddead6736161f80f497699780e0f2c92c50edb1d938eb025f95c" exitCode=0
Oct 31 16:35:11.745558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.745422    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" event={"ID":"93f57064-04ed-4fb1-98be-d49e9911384c","Type":"ContainerDied","Data":"215e0ab0b759ddead6736161f80f497699780e0f2c92c50edb1d938eb025f95c"}
Oct 31 16:35:11.942431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.942059    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:35:11.991334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:11.991310    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:35:12.032112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.032081    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\" (UniqueName: \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\") pod \"93f57064-04ed-4fb1-98be-d49e9911384c\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") "
Oct 31 16:35:12.032606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.032280    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\") pod \"93f57064-04ed-4fb1-98be-d49e9911384c\" (UID: \"93f57064-04ed-4fb1-98be-d49e9911384c\") "
Oct 31 16:35:12.032731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.032705    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume" pod "93f57064-04ed-4fb1-98be-d49e9911384c" (UID: "93f57064-04ed-4fb1-98be-d49e9911384c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:12.033022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.033001    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93f57064-04ed-4fb1-98be-d49e9911384c-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:12.034279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.034250    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw" pod "93f57064-04ed-4fb1-98be-d49e9911384c" (UID: "93f57064-04ed-4fb1-98be-d49e9911384c"). InnerVolumeSpecName "kube-api-access-k77xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:12.134237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.134142    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\" (UniqueName: \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\") pod \"b4bbbab1-405f-4534-bce0-76ab238a4569\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") "
Oct 31 16:35:12.134406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.134334    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\") pod \"b4bbbab1-405f-4534-bce0-76ab238a4569\" (UID: \"b4bbbab1-405f-4534-bce0-76ab238a4569\") "
Oct 31 16:35:12.134804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.134775    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume" pod "b4bbbab1-405f-4534-bce0-76ab238a4569" (UID: "b4bbbab1-405f-4534-bce0-76ab238a4569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:12.135095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.135071    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77xw\" (UniqueName: \"kubernetes.io/projected/93f57064-04ed-4fb1-98be-d49e9911384c-kube-api-access-k77xw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:12.135274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.135252    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bbbab1-405f-4534-bce0-76ab238a4569-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:12.136719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.136698    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms" pod "b4bbbab1-405f-4534-bce0-76ab238a4569" (UID: "b4bbbab1-405f-4534-bce0-76ab238a4569"). InnerVolumeSpecName "kube-api-access-2rzms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:12.236553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.236513    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzms\" (UniqueName: \"kubernetes.io/projected/b4bbbab1-405f-4534-bce0-76ab238a4569-kube-api-access-2rzms\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:12.778151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.777354    4417 generic.go:356] "Generic (PLEG): container finished" podID="b4bbbab1-405f-4534-bce0-76ab238a4569" containerID="42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd" exitCode=0
Oct 31 16:35:12.778151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.777545    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" event={"ID":"b4bbbab1-405f-4534-bce0-76ab238a4569","Type":"ContainerDied","Data":"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"}
Oct 31 16:35:12.778151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.777704    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9" event={"ID":"b4bbbab1-405f-4534-bce0-76ab238a4569","Type":"ContainerDied","Data":"61a761047286c3bf8c180159eaaace3112e2453b6dac643d689be2ad91ec791c"}
Oct 31 16:35:12.778151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.777743    4417 scope.go:122] "RemoveContainer" containerID="42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"
Oct 31 16:35:12.778151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.777923    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"
Oct 31 16:35:12.787027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.786998    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f" event={"ID":"93f57064-04ed-4fb1-98be-d49e9911384c","Type":"ContainerDied","Data":"3d5e66c80431e5ad73e8ecda253bd76c34c7478efc68c44c2734d28f4bc7761e"}
Oct 31 16:35:12.787208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.787019    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"
Oct 31 16:35:12.796509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.796429    4417 scope.go:122] "RemoveContainer" containerID="42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"
Oct 31 16:35:12.797091 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:12.796910    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd\": container with ID starting with 42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd not found: ID does not exist" containerID="42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"
Oct 31 16:35:12.797091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.796954    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd"} err="failed to get container status \"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd\": rpc error: code = NotFound desc = could not find container \"42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd\": container with ID starting with 42dee7c320d04b793d5b9b5beb2c97cc449c0329b2dc1e80f9e5d11c96cba9cd not found: ID does not exist"
Oct 31 16:35:12.797091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.796988    4417 scope.go:122] "RemoveContainer" containerID="215e0ab0b759ddead6736161f80f497699780e0f2c92c50edb1d938eb025f95c"
Oct 31 16:35:12.946262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.946161    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:35:12.949849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:12.949817    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6b1cca-5c4798f5cc-7mt2f"]
Oct 31 16:35:13.044318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:13.041396    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:35:13.051841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:13.051801    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f57064-04ed-4fb1-98be-d49e9911384c" path="/var/lib/kubelet/pods/93f57064-04ed-4fb1-98be-d49e9911384c/volumes"
Oct 31 16:35:13.052149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:13.052133    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6b1cca-5f6fcfc8c9-g5hl9"]
Oct 31 16:35:15.053087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:15.053056    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bbbab1-405f-4534-bce0-76ab238a4569" path="/var/lib/kubelet/pods/b4bbbab1-405f-4534-bce0-76ab238a4569/volumes"
Oct 31 16:35:15.691884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:15.691848    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:35:15.792134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:15.792103    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:35:16.010324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.010235    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" podUID="6de4c19b-ad77-407d-be89-6ac94284fe85" containerName="payload-processing" containerID="cri-o://4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2" gracePeriod=25
Oct 31 16:35:16.108603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.108567    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" podUID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" containerName="payload-pre-processing" containerID="cri-o://caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5" gracePeriod=25
Oct 31 16:35:16.421533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.421503    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:35:16.508884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.508861    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:35:16.522101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522075    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\" (UniqueName: \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\") pod \"6de4c19b-ad77-407d-be89-6ac94284fe85\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") "
Oct 31 16:35:16.522226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522123    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\" (UniqueName: \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\") pod \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") "
Oct 31 16:35:16.522226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522162    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\") pod \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\" (UID: \"515acf0c-2a58-44f1-92bc-ae872d52dd3a\") "
Oct 31 16:35:16.522319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522233    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\") pod \"6de4c19b-ad77-407d-be89-6ac94284fe85\" (UID: \"6de4c19b-ad77-407d-be89-6ac94284fe85\") "
Oct 31 16:35:16.522570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522544    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume" pod "515acf0c-2a58-44f1-92bc-ae872d52dd3a" (UID: "515acf0c-2a58-44f1-92bc-ae872d52dd3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:16.522676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522605    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume" pod "6de4c19b-ad77-407d-be89-6ac94284fe85" (UID: "6de4c19b-ad77-407d-be89-6ac94284fe85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:16.522850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522831    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515acf0c-2a58-44f1-92bc-ae872d52dd3a-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:16.522917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.522855    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de4c19b-ad77-407d-be89-6ac94284fe85-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:16.524473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.524450    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p" pod "515acf0c-2a58-44f1-92bc-ae872d52dd3a" (UID: "515acf0c-2a58-44f1-92bc-ae872d52dd3a"). InnerVolumeSpecName "kube-api-access-z8q7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:16.524605 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.524589    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg" pod "6de4c19b-ad77-407d-be89-6ac94284fe85" (UID: "6de4c19b-ad77-407d-be89-6ac94284fe85"). InnerVolumeSpecName "kube-api-access-lxmbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:16.623725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.623645    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmbg\" (UniqueName: \"kubernetes.io/projected/6de4c19b-ad77-407d-be89-6ac94284fe85-kube-api-access-lxmbg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:16.623725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.623675    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8q7p\" (UniqueName: \"kubernetes.io/projected/515acf0c-2a58-44f1-92bc-ae872d52dd3a-kube-api-access-z8q7p\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:16.821109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.821077    4417 generic.go:356] "Generic (PLEG): container finished" podID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" containerID="caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5" exitCode=0
Oct 31 16:35:16.821284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.821121    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"
Oct 31 16:35:16.821284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.821158    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" event={"ID":"515acf0c-2a58-44f1-92bc-ae872d52dd3a","Type":"ContainerDied","Data":"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"}
Oct 31 16:35:16.821284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.821205    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc" event={"ID":"515acf0c-2a58-44f1-92bc-ae872d52dd3a","Type":"ContainerDied","Data":"2b60e76e0597d2e6339d66df1e7e2fe0fec9d7d07904a1a824d28b4b442ea300"}
Oct 31 16:35:16.821284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.821231    4417 scope.go:122] "RemoveContainer" containerID="caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"
Oct 31 16:35:16.822782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.822760    4417 generic.go:356] "Generic (PLEG): container finished" podID="6de4c19b-ad77-407d-be89-6ac94284fe85" containerID="4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2" exitCode=0
Oct 31 16:35:16.822906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.822802    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"
Oct 31 16:35:16.822906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.822830    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" event={"ID":"6de4c19b-ad77-407d-be89-6ac94284fe85","Type":"ContainerDied","Data":"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"}
Oct 31 16:35:16.822906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.822886    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk" event={"ID":"6de4c19b-ad77-407d-be89-6ac94284fe85","Type":"ContainerDied","Data":"1ddd0b05f2a2fef483730c9f6f43af0469a62136553b2833000dab36001925b2"}
Oct 31 16:35:16.832540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.832517    4417 scope.go:122] "RemoveContainer" containerID="caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"
Oct 31 16:35:16.832861 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:16.832839    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5\": container with ID starting with caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5 not found: ID does not exist" containerID="caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"
Oct 31 16:35:16.832944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.832872    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5"} err="failed to get container status \"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5\": rpc error: code = NotFound desc = could not find container \"caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5\": container with ID starting with caea76bcdf02caae531612081211f8d9b6831e4f933b9071349f0a86e676bee5 not found: ID does not exist"
Oct 31 16:35:16.832944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.832895    4417 scope.go:122] "RemoveContainer" containerID="4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"
Oct 31 16:35:16.844541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.844519    4417 scope.go:122] "RemoveContainer" containerID="4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"
Oct 31 16:35:16.844909 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:16.844880    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2\": container with ID starting with 4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2 not found: ID does not exist" containerID="4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"
Oct 31 16:35:16.844972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.844922    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2"} err="failed to get container status \"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2\": rpc error: code = NotFound desc = could not find container \"4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2\": container with ID starting with 4be392102e93ea62f6ff516ab9ec4c589164164029538f6b074c2c3918f44db2 not found: ID does not exist"
Oct 31 16:35:16.974244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.974155    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:35:16.978132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:16.978103    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-822dd4-74865cd46c-7kbdc"]
Oct 31 16:35:17.056635 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:17.056584    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" path="/var/lib/kubelet/pods/515acf0c-2a58-44f1-92bc-ae872d52dd3a/volumes"
Oct 31 16:35:17.078147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:17.078116    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:35:17.083425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:17.083399    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-822dd4-666b4c4c69-87ntk"]
Oct 31 16:35:19.051296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:19.051264    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de4c19b-ad77-407d-be89-6ac94284fe85" path="/var/lib/kubelet/pods/6de4c19b-ad77-407d-be89-6ac94284fe85/volumes"
Oct 31 16:35:20.692868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:20.692827    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" podUID="56e77561-0a8c-4e83-9524-187cef29f604" containerName="payload-processing" containerID="cri-o://4559efb64a26ea52355cd4e5336b300869433392a6f5cc7395a3c0179d8c4d40" gracePeriod=25
Oct 31 16:35:20.792954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:20.792914    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" podUID="c0e7c86e-1e1d-4261-b953-193b3c066923" containerName="payload-pre-processing" containerID="cri-o://b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895" gracePeriod=25
Oct 31 16:35:20.855832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:20.855800    4417 generic.go:356] "Generic (PLEG): container finished" podID="56e77561-0a8c-4e83-9524-187cef29f604" containerID="4559efb64a26ea52355cd4e5336b300869433392a6f5cc7395a3c0179d8c4d40" exitCode=0
Oct 31 16:35:20.855996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:20.855873    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" event={"ID":"56e77561-0a8c-4e83-9524-187cef29f604","Type":"ContainerDied","Data":"4559efb64a26ea52355cd4e5336b300869433392a6f5cc7395a3c0179d8c4d40"}
Oct 31 16:35:21.017561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.017506    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:35:21.056462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.056428    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\") pod \"56e77561-0a8c-4e83-9524-187cef29f604\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") "
Oct 31 16:35:21.056850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.056823    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume" pod "56e77561-0a8c-4e83-9524-187cef29f604" (UID: "56e77561-0a8c-4e83-9524-187cef29f604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:21.057018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.056997    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\" (UniqueName: \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\") pod \"56e77561-0a8c-4e83-9524-187cef29f604\" (UID: \"56e77561-0a8c-4e83-9524-187cef29f604\") "
Oct 31 16:35:21.059702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.059680    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e77561-0a8c-4e83-9524-187cef29f604-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:21.059958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.059936    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs" pod "56e77561-0a8c-4e83-9524-187cef29f604" (UID: "56e77561-0a8c-4e83-9524-187cef29f604"). InnerVolumeSpecName "kube-api-access-gfrcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:21.089000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.088974    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:35:21.160861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.160828    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\" (UniqueName: \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\") pod \"c0e7c86e-1e1d-4261-b953-193b3c066923\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") "
Oct 31 16:35:21.161043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.160896    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\") pod \"c0e7c86e-1e1d-4261-b953-193b3c066923\" (UID: \"c0e7c86e-1e1d-4261-b953-193b3c066923\") "
Oct 31 16:35:21.161380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.161348    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume" pod "c0e7c86e-1e1d-4261-b953-193b3c066923" (UID: "c0e7c86e-1e1d-4261-b953-193b3c066923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:21.161600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.161577    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e7c86e-1e1d-4261-b953-193b3c066923-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:21.161721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.161604    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfrcs\" (UniqueName: \"kubernetes.io/projected/56e77561-0a8c-4e83-9524-187cef29f604-kube-api-access-gfrcs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:21.163589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.163564    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7" pod "c0e7c86e-1e1d-4261-b953-193b3c066923" (UID: "c0e7c86e-1e1d-4261-b953-193b3c066923"). InnerVolumeSpecName "kube-api-access-b8mz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:21.263485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.263403    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mz7\" (UniqueName: \"kubernetes.io/projected/c0e7c86e-1e1d-4261-b953-193b3c066923-kube-api-access-b8mz7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:21.863460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.863422    4417 generic.go:356] "Generic (PLEG): container finished" podID="c0e7c86e-1e1d-4261-b953-193b3c066923" containerID="b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895" exitCode=0
Oct 31 16:35:21.863959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.863469    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"
Oct 31 16:35:21.863959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.863513    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" event={"ID":"c0e7c86e-1e1d-4261-b953-193b3c066923","Type":"ContainerDied","Data":"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"}
Oct 31 16:35:21.863959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.863595    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8" event={"ID":"c0e7c86e-1e1d-4261-b953-193b3c066923","Type":"ContainerDied","Data":"c3d64ea5baaadf275fa1a8b059008fc29dec657a436a013ee33d0f5ef1f9c2d1"}
Oct 31 16:35:21.863959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.863638    4417 scope.go:122] "RemoveContainer" containerID="b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"
Oct 31 16:35:21.865292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.865274    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"
Oct 31 16:35:21.865292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.865279    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76" event={"ID":"56e77561-0a8c-4e83-9524-187cef29f604","Type":"ContainerDied","Data":"e4626808523da5603b3dd830b317ee18a43cd8aa01d7cda43bc8d0bf4a2e7245"}
Oct 31 16:35:21.875256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.875239    4417 scope.go:122] "RemoveContainer" containerID="b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"
Oct 31 16:35:21.875532 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:21.875508    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895\": container with ID starting with b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895 not found: ID does not exist" containerID="b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"
Oct 31 16:35:21.875682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.875541    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895"} err="failed to get container status \"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895\": rpc error: code = NotFound desc = could not find container \"b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895\": container with ID starting with b02c9c39caa90385dc925935d39ce284aab4efb274b3cea193b1d3e37b181895 not found: ID does not exist"
Oct 31 16:35:21.875682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:21.875563    4417 scope.go:122] "RemoveContainer" containerID="4559efb64a26ea52355cd4e5336b300869433392a6f5cc7395a3c0179d8c4d40"
Oct 31 16:35:22.021470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:22.021318    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:35:22.025183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:22.025155    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dac4fa-6dc74cb485-skh76"]
Oct 31 16:35:22.125169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:22.125137    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:35:22.128626 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:22.128591    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dac4fa-576669c89f-wqcw8"]
Oct 31 16:35:23.054097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:23.054062    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e77561-0a8c-4e83-9524-187cef29f604" path="/var/lib/kubelet/pods/56e77561-0a8c-4e83-9524-187cef29f604/volumes"
Oct 31 16:35:23.054696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:23.054679    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e7c86e-1e1d-4261-b953-193b3c066923" path="/var/lib/kubelet/pods/c0e7c86e-1e1d-4261-b953-193b3c066923/volumes"
Oct 31 16:35:24.020641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.013262    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:35:24.020641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.013549    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" podUID="f2b1c3cb-a618-4329-8772-774c3d5b110c" containerName="maas-api" containerID="cri-o://c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1" gracePeriod=30
Oct 31 16:35:24.454153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.454125    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:35:24.566673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.556842    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\") pod \"f2b1c3cb-a618-4329-8772-774c3d5b110c\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") "
Oct 31 16:35:24.566673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.557085    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\") pod \"f2b1c3cb-a618-4329-8772-774c3d5b110c\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") "
Oct 31 16:35:24.566673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.557422    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\") pod \"f2b1c3cb-a618-4329-8772-774c3d5b110c\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") "
Oct 31 16:35:24.566673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.557542    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\") pod \"f2b1c3cb-a618-4329-8772-774c3d5b110c\" (UID: \"f2b1c3cb-a618-4329-8772-774c3d5b110c\") "
Oct 31 16:35:24.566673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.564321    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca" pod "f2b1c3cb-a618-4329-8772-774c3d5b110c" (UID: "f2b1c3cb-a618-4329-8772-774c3d5b110c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:24.602408 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.601937    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls" pod "f2b1c3cb-a618-4329-8772-774c3d5b110c" (UID: "f2b1c3cb-a618-4329-8772-774c3d5b110c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:24.602875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.602837    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs" pod "f2b1c3cb-a618-4329-8772-774c3d5b110c" (UID: "f2b1c3cb-a618-4329-8772-774c3d5b110c"). InnerVolumeSpecName "kube-api-access-6bbqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:24.603183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.603146    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs" pod "f2b1c3cb-a618-4329-8772-774c3d5b110c" (UID: "f2b1c3cb-a618-4329-8772-774c3d5b110c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:24.660088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.660055    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/f2b1c3cb-a618-4329-8772-774c3d5b110c-kube-api-access-6bbqs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:24.660274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.660100    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:24.660274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.660114    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b1c3cb-a618-4329-8772-774c3d5b110c-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:24.660274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.660123    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b1c3cb-a618-4329-8772-774c3d5b110c-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:24.889193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.889156    4417 generic.go:356] "Generic (PLEG): container finished" podID="f2b1c3cb-a618-4329-8772-774c3d5b110c" containerID="c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1" exitCode=0
Oct 31 16:35:24.889390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.889197    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"
Oct 31 16:35:24.889390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.889239    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" event={"ID":"f2b1c3cb-a618-4329-8772-774c3d5b110c","Type":"ContainerDied","Data":"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"}
Oct 31 16:35:24.889390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.889293    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26" event={"ID":"f2b1c3cb-a618-4329-8772-774c3d5b110c","Type":"ContainerDied","Data":"98e7795439aadcefea569bfe34204237b10a534fd7bdb0fb4a413d25f2d05c97"}
Oct 31 16:35:24.889390 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.889321    4417 scope.go:122] "RemoveContainer" containerID="c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"
Oct 31 16:35:24.901045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.901025    4417 scope.go:122] "RemoveContainer" containerID="c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"
Oct 31 16:35:24.901327 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:24.901306    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1\": container with ID starting with c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1 not found: ID does not exist" containerID="c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"
Oct 31 16:35:24.901399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:24.901334    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1"} err="failed to get container status \"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1\": rpc error: code = NotFound desc = could not find container \"c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1\": container with ID starting with c603d8209952fe666ff6bf4abee124c310a175cf489ad478bf7a5038c2a417e1 not found: ID does not exist"
Oct 31 16:35:25.183574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:25.183491    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:35:25.183574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:25.183544    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e977aa-d8948f57f-5mg26"]
Oct 31 16:35:27.054452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:27.054419    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b1c3cb-a618-4329-8772-774c3d5b110c" path="/var/lib/kubelet/pods/f2b1c3cb-a618-4329-8772-774c3d5b110c/volumes"
Oct 31 16:35:28.757822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:28.757784    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:35:32.794031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:32.793996    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:35:32.794545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:32.794216    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy" containerID="cri-o://812026b273c375c3301e032bf35c1ef4fa6cb6d0134fa73b2e84e9decfe8ea8d" gracePeriod=30
Oct 31 16:35:33.068698 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:33.068594    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:35:33.758596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:33.758556    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" podUID="2c3119ac-4414-4f7f-94db-8478555dc8d5" containerName="payload-pre-processing" containerID="cri-o://7aea6d34e456a5112a7977dc2a565b61d6ea2621a2933c3d03fab191c8ebb996" gracePeriod=25
Oct 31 16:35:33.999873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:33.999712    4417 generic.go:356] "Generic (PLEG): container finished" podID="2c3119ac-4414-4f7f-94db-8478555dc8d5" containerID="7aea6d34e456a5112a7977dc2a565b61d6ea2621a2933c3d03fab191c8ebb996" exitCode=0
Oct 31 16:35:33.999873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:33.999780    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" event={"ID":"2c3119ac-4414-4f7f-94db-8478555dc8d5","Type":"ContainerDied","Data":"7aea6d34e456a5112a7977dc2a565b61d6ea2621a2933c3d03fab191c8ebb996"}
Oct 31 16:35:34.045531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.045501    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:35:34.147104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.147073    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\") pod \"2c3119ac-4414-4f7f-94db-8478555dc8d5\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") "
Oct 31 16:35:34.147310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.147206    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\" (UniqueName: \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\") pod \"2c3119ac-4414-4f7f-94db-8478555dc8d5\" (UID: \"2c3119ac-4414-4f7f-94db-8478555dc8d5\") "
Oct 31 16:35:34.147517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.147490    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume" pod "2c3119ac-4414-4f7f-94db-8478555dc8d5" (UID: "2c3119ac-4414-4f7f-94db-8478555dc8d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:34.147745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.147729    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3119ac-4414-4f7f-94db-8478555dc8d5-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:34.149579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.149540    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx" pod "2c3119ac-4414-4f7f-94db-8478555dc8d5" (UID: "2c3119ac-4414-4f7f-94db-8478555dc8d5"). InnerVolumeSpecName "kube-api-access-8z9gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:34.249229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:34.249194    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z9gx\" (UniqueName: \"kubernetes.io/projected/2c3119ac-4414-4f7f-94db-8478555dc8d5-kube-api-access-8z9gx\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:35.008329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:35.008304    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"
Oct 31 16:35:35.008816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:35.008302    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7" event={"ID":"2c3119ac-4414-4f7f-94db-8478555dc8d5","Type":"ContainerDied","Data":"1048628fbfb2dc1f062d5e4a68c87a513e0c97ed87dfb8b91a3a6142a591d0bd"}
Oct 31 16:35:35.008816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:35.008440    4417 scope.go:122] "RemoveContainer" containerID="7aea6d34e456a5112a7977dc2a565b61d6ea2621a2933c3d03fab191c8ebb996"
Oct 31 16:35:35.162492 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:35.162456    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:35:35.206051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:35.206020    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdc828-546768c7f5-vjkp7"]
Oct 31 16:35:36.022531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.022501    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:35:36.023402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023382    4417 cpu_manager.go:414] "containerMap: removing container" podUID="2c3119ac-4414-4f7f-94db-8478555dc8d5" containerName="payload-pre-processing"
Oct 31 16:35:36.023402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023401    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3119ac-4414-4f7f-94db-8478555dc8d5" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023415    4417 cpu_manager.go:414] "containerMap: removing container" podUID="f2b1c3cb-a618-4329-8772-774c3d5b110c" containerName="maas-api"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023420    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b1c3cb-a618-4329-8772-774c3d5b110c" containerName="maas-api"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023438    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023443    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023451    4417 cpu_manager.go:414] "containerMap: removing container" podUID="93f57064-04ed-4fb1-98be-d49e9911384c" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023456    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f57064-04ed-4fb1-98be-d49e9911384c" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023465    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023469    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023479    4417 cpu_manager.go:414] "containerMap: removing container" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023483    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023496    4417 cpu_manager.go:414] "containerMap: removing container" podUID="56e77561-0a8c-4e83-9524-187cef29f604" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023502    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e77561-0a8c-4e83-9524-187cef29f604" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023513    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6de4c19b-ad77-407d-be89-6ac94284fe85" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023518    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de4c19b-ad77-407d-be89-6ac94284fe85" containerName="payload-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023540    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c0e7c86e-1e1d-4261-b953-193b3c066923" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023545    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e7c86e-1e1d-4261-b953-193b3c066923" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023554    4417 cpu_manager.go:414] "containerMap: removing container" podUID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023558    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023565    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b4bbbab1-405f-4534-bce0-76ab238a4569" containerName="payload-pre-processing"
Oct 31 16:35:36.023569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023570    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bbbab1-405f-4534-bce0-76ab238a4569" containerName="payload-pre-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023682    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="515acf0c-2a58-44f1-92bc-ae872d52dd3a" containerName="payload-pre-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023693    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bbbab1-405f-4534-bce0-76ab238a4569" containerName="payload-pre-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023701    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3119ac-4414-4f7f-94db-8478555dc8d5" containerName="payload-pre-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023710    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b1c3cb-a618-4329-8772-774c3d5b110c" containerName="maas-api"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023717    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1d880b-ff80-4de9-b6ec-eef487720152" containerName="istio-proxy"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023723    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f57064-04ed-4fb1-98be-d49e9911384c" containerName="payload-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023728    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8191271-030d-4170-b05e-ac9c285d8f4d" containerName="istio-proxy"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023735    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc1ac91-b961-4fd1-b498-3c774e480a17" containerName="istio-proxy"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023741    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e77561-0a8c-4e83-9524-187cef29f604" containerName="payload-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023747    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de4c19b-ad77-407d-be89-6ac94284fe85" containerName="payload-processing"
Oct 31 16:35:36.024593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.023756    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e7c86e-1e1d-4261-b953-193b3c066923" containerName="payload-pre-processing"
Oct 31 16:35:36.028783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.028766    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.077144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.077122    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-05db63-openshift-default-dockercfg-qccxp\""
Oct 31 16:35:36.079384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079360    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mhh\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079478    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079524    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079594    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079663    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079749    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079840    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079876    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.079967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.079936    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.127265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.127226    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:35:36.254193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.254061    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.254939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.254837    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.254939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.254917    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255021    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mhh\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255095    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255129    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255202    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255255    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.255359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.255352    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.257056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.256169    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.260177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.258026    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.260348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.259891    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.260599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.260478    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.260599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.260560    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.262204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.262173    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.264917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.264884    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.309971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.309947    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.312227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.312207    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mhh\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\") pod \"e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") " pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.348019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.347995    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:36.798845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.796785    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:35:36.801727 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:35:36.801584    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f40a63_3afb_456c_a3d6_0ad3d9296e3a.slice/crio-a3544b373305936c1cf0a6368dc42512faf7e8f36781f88c6f4f4aeee1be1a22 WatchSource:0}: Error finding container a3544b373305936c1cf0a6368dc42512faf7e8f36781f88c6f4f4aeee1be1a22: Status 404 returned error can't find the container with id a3544b373305936c1cf0a6368dc42512faf7e8f36781f88c6f4f4aeee1be1a22
Oct 31 16:35:36.805664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.805592    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:36.805781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.805716    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:36.805781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:36.805766    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:37.024488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.024449    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerStarted","Data":"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"}
Oct 31 16:35:37.024916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.024520    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerStarted","Data":"a3544b373305936c1cf0a6368dc42512faf7e8f36781f88c6f4f4aeee1be1a22"}
Oct 31 16:35:37.055825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.055746    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3119ac-4414-4f7f-94db-8478555dc8d5" path="/var/lib/kubelet/pods/2c3119ac-4414-4f7f-94db-8478555dc8d5/volumes"
Oct 31 16:35:37.136339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.136273    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podStartSLOduration=2.136253179 podStartE2EDuration="2.136253179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:37.131166685 +0000 UTC m=+1982.192741870" watchObservedRunningTime="2026-10-31 16:35:37.136253179 +0000 UTC m=+1982.197828365"
Oct 31 16:35:37.353474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.353386    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:37.356223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.355116    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:35:37.356223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:37.355172    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:35:38.036588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.036552    4417 generic.go:356] "Generic (PLEG): container finished" podID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerID="812026b273c375c3301e032bf35c1ef4fa6cb6d0134fa73b2e84e9decfe8ea8d" exitCode=0
Oct 31 16:35:38.037583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.037557    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerDied","Data":"812026b273c375c3301e032bf35c1ef4fa6cb6d0134fa73b2e84e9decfe8ea8d"}
Oct 31 16:35:38.037705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.037634    4417 scope.go:122] "RemoveContainer" containerID="2124dba75bb820e75ee3c83236da1ae89debddc4f0515779aebc8f4d11baef0c"
Oct 31 16:35:38.092366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.092342    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:35:38.103751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103725    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.103854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103786    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.103854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103833    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.103964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103888    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.103964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103911    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.103964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.103940    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.104122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104005    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.104122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104077    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.104237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104128    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\") pod \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\" (UID: \"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be\") "
Oct 31 16:35:38.104237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104194    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:38.104237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104210    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:38.104553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104531    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:38.104654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104548    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:38.104773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104753    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.104843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104776    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.104843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104808    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.104843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.104828    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.105037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.105011    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:38.106306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.106257    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:38.106433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.106342    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:35:38.106433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.106379    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "kube-api-access-kqmsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:38.106659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.106641    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy" pod "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" (UID: "4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:38.206050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.206010    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.206050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.206041    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.206050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.206055    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmsz\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-kube-api-access-kqmsz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.206315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.206069    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.206315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.206081    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:38.353753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:38.353662    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:39.048146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.048114    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"
Oct 31 16:35:39.052757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.052731    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:39.052923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.052780    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9" event={"ID":"4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be","Type":"ContainerDied","Data":"f0b8f845268a94b6e70a0f6960ff648d11023c1a1abe37c43abf5fd61b33b7a8"}
Oct 31 16:35:39.052923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.052845    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:35:39.052923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.052873    4417 scope.go:122] "RemoveContainer" containerID="812026b273c375c3301e032bf35c1ef4fa6cb6d0134fa73b2e84e9decfe8ea8d"
Oct 31 16:35:39.357303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.357210    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:35:39.361933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:39.361902    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e977aa-openshift-default-67bb57797-kfhv9"]
Oct 31 16:35:41.056073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.056040    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" path="/var/lib/kubelet/pods/4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be/volumes"
Oct 31 16:35:41.212709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.212676    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:35:41.212941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.212888    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" podUID="439a64af-1645-4690-bc3b-1b5cf856c432" containerName="maas-api" containerID="cri-o://8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1" gracePeriod=30
Oct 31 16:35:41.591427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.591350    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:35:41.665884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.665842    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") pod \"439a64af-1645-4690-bc3b-1b5cf856c432\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") "
Oct 31 16:35:41.666115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.666098    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\") pod \"439a64af-1645-4690-bc3b-1b5cf856c432\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") "
Oct 31 16:35:41.666214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.666198    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\") pod \"439a64af-1645-4690-bc3b-1b5cf856c432\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") "
Oct 31 16:35:41.666333 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.666260    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\" (UniqueName: \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\") pod \"439a64af-1645-4690-bc3b-1b5cf856c432\" (UID: \"439a64af-1645-4690-bc3b-1b5cf856c432\") "
Oct 31 16:35:41.666551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.666528    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca" pod "439a64af-1645-4690-bc3b-1b5cf856c432" (UID: "439a64af-1645-4690-bc3b-1b5cf856c432"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:41.666918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.666893    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/439a64af-1645-4690-bc3b-1b5cf856c432-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:41.668191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.668142    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls" pod "439a64af-1645-4690-bc3b-1b5cf856c432" (UID: "439a64af-1645-4690-bc3b-1b5cf856c432"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:41.668862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.668842    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl" pod "439a64af-1645-4690-bc3b-1b5cf856c432" (UID: "439a64af-1645-4690-bc3b-1b5cf856c432"). InnerVolumeSpecName "kube-api-access-xprrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:41.669151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.669135    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs" pod "439a64af-1645-4690-bc3b-1b5cf856c432" (UID: "439a64af-1645-4690-bc3b-1b5cf856c432"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:41.786388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.786177    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:41.786388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.786212    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprrl\" (UniqueName: \"kubernetes.io/projected/439a64af-1645-4690-bc3b-1b5cf856c432-kube-api-access-xprrl\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:41.786388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:41.786229    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/439a64af-1645-4690-bc3b-1b5cf856c432-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:42.075938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.075908    4417 generic.go:356] "Generic (PLEG): container finished" podID="439a64af-1645-4690-bc3b-1b5cf856c432" containerID="8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1" exitCode=0
Oct 31 16:35:42.076360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.075969    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"
Oct 31 16:35:42.076360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.075987    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" event={"ID":"439a64af-1645-4690-bc3b-1b5cf856c432","Type":"ContainerDied","Data":"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"}
Oct 31 16:35:42.076360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.076045    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g" event={"ID":"439a64af-1645-4690-bc3b-1b5cf856c432","Type":"ContainerDied","Data":"3bbf4992005505450bb6d396b14a8b1776a3fac9d11774f535336e08d59bde5a"}
Oct 31 16:35:42.076360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.076104    4417 scope.go:122] "RemoveContainer" containerID="8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"
Oct 31 16:35:42.097059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.097025    4417 scope.go:122] "RemoveContainer" containerID="8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"
Oct 31 16:35:42.097385 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:42.097362    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1\": container with ID starting with 8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1 not found: ID does not exist" containerID="8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"
Oct 31 16:35:42.097428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.097394    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1"} err="failed to get container status \"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1\": rpc error: code = NotFound desc = could not find container \"8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1\": container with ID starting with 8ef922a77ec17f7a197f658cea6f8cef02effc48146c305027e3e82f5043a8d1 not found: ID does not exist"
Oct 31 16:35:42.229117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.229079    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:35:42.274716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:42.274682    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdc828-8d67fcdc4-zz84g"]
Oct 31 16:35:43.061026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.060990    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439a64af-1645-4690-bc3b-1b5cf856c432" path="/var/lib/kubelet/pods/439a64af-1645-4690-bc3b-1b5cf856c432/volumes"
Oct 31 16:35:43.516709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.516573    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:35:43.518791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.518762    4417 cpu_manager.go:414] "containerMap: removing container" podUID="439a64af-1645-4690-bc3b-1b5cf856c432" containerName="maas-api"
Oct 31 16:35:43.518990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.518968    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439a64af-1645-4690-bc3b-1b5cf856c432" containerName="maas-api"
Oct 31 16:35:43.519106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.519094    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.519204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.519189    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.519516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.519499    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="439a64af-1645-4690-bc3b-1b5cf856c432" containerName="maas-api"
Oct 31 16:35:43.519698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.519685    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.519821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.519810    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.539427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.539398    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.574132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.574101    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:35:43.663830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.663784    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.664149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.664112    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttqn\" (UniqueName: \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.690637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.665071    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-05db63-dockercfg-bbq75\""
Oct 31 16:35:43.690884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.673390    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-05db63\""
Oct 31 16:35:43.758897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.757980    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:35:43.759232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.759206    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.759318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.759232    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec8e8c4-aa66-4e8e-bfda-f1f07ad3b1be" containerName="istio-proxy"
Oct 31 16:35:43.770346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.764406    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:43.778059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.778027    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.778300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.778281    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttqn\" (UniqueName: \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.779933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.779910    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.810843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.810810    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:35:43.832098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.832061    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttqn\" (UniqueName: \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\") pod \"payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.856595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.856561    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:43.879629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.879587    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2nc\" (UniqueName: \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:43.879830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.879812    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:43.982647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.981919    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2nc\" (UniqueName: \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:43.982647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.982134    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:43.984685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:43.983166    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:44.037700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:44.036940    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2nc\" (UniqueName: \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\") pod \"payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:44.107249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:44.107216    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:44.300105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:44.300074    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:35:44.301463 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:35:44.301427    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8623aa8e_31c6_4588_bf69_42772ad070b5.slice/crio-4ca42df58aceac8489719efc9bf63031a12aa9087e10d9dda8543a98980661fe WatchSource:0}: Error finding container 4ca42df58aceac8489719efc9bf63031a12aa9087e10d9dda8543a98980661fe: Status 404 returned error can't find the container with id 4ca42df58aceac8489719efc9bf63031a12aa9087e10d9dda8543a98980661fe
Oct 31 16:35:44.482425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:44.482401    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:35:44.484176 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:35:44.484149    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0168c750_b6e1_415d_8118_906952bd613d.slice/crio-1b8b9ab654ee54451231f73ee07ff3f2340d86641b8511b50efb1016e8586240 WatchSource:0}: Error finding container 1b8b9ab654ee54451231f73ee07ff3f2340d86641b8511b50efb1016e8586240: Status 404 returned error can't find the container with id 1b8b9ab654ee54451231f73ee07ff3f2340d86641b8511b50efb1016e8586240
Oct 31 16:35:45.101391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.101356    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" event={"ID":"0168c750-b6e1-415d-8118-906952bd613d","Type":"ContainerStarted","Data":"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"}
Oct 31 16:35:45.101865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.101410    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" event={"ID":"0168c750-b6e1-415d-8118-906952bd613d","Type":"ContainerStarted","Data":"1b8b9ab654ee54451231f73ee07ff3f2340d86641b8511b50efb1016e8586240"}
Oct 31 16:35:45.101865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.101434    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:45.103102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.103078    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" event={"ID":"8623aa8e-31c6-4588-bf69-42772ad070b5","Type":"ContainerStarted","Data":"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"}
Oct 31 16:35:45.103201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.103117    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" event={"ID":"8623aa8e-31c6-4588-bf69-42772ad070b5","Type":"ContainerStarted","Data":"4ca42df58aceac8489719efc9bf63031a12aa9087e10d9dda8543a98980661fe"}
Oct 31 16:35:45.103201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.103155    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:45.207997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.207950    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" podStartSLOduration=2.207935678 podStartE2EDuration="2.207935678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:45.205846123 +0000 UTC m=+1990.267421311" watchObservedRunningTime="2026-10-31 16:35:45.207935678 +0000 UTC m=+1990.269510864"
Oct 31 16:35:45.312936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:45.312883    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" podStartSLOduration=2.312869417 podStartE2EDuration="2.312869417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:45.311510984 +0000 UTC m=+1990.373086167" watchObservedRunningTime="2026-10-31 16:35:45.312869417 +0000 UTC m=+1990.374444602"
Oct 31 16:35:50.530661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:50.530609    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:35:50.531092 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:50.530839    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" containerID="cri-o://a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22" gracePeriod=30
Oct 31 16:35:52.652907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:52.652870    4417 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 16:35:52.653309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:52.652928    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:35:52.847107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:52.847076    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:35:52.847299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:52.847256    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" podUID="11daf9e6-002c-444d-88d8-1f666b55ce84" containerName="maas-api" containerID="cri-o://5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484" gracePeriod=30
Oct 31 16:35:53.120246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.120210    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:35:53.153751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.153711    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:35:53.155042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.155013    4417 cpu_manager.go:414] "containerMap: removing container" podUID="11daf9e6-002c-444d-88d8-1f666b55ce84" containerName="maas-api"
Oct 31 16:35:53.155042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.155039    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11daf9e6-002c-444d-88d8-1f666b55ce84" containerName="maas-api"
Oct 31 16:35:53.155232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.155171    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="11daf9e6-002c-444d-88d8-1f666b55ce84" containerName="maas-api"
Oct 31 16:35:53.158815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.158796    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.202860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.202823    4417 generic.go:356] "Generic (PLEG): container finished" podID="11daf9e6-002c-444d-88d8-1f666b55ce84" containerID="5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484" exitCode=0
Oct 31 16:35:53.203041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.202868    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" event={"ID":"11daf9e6-002c-444d-88d8-1f666b55ce84","Type":"ContainerDied","Data":"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"}
Oct 31 16:35:53.203041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.202931    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x" event={"ID":"11daf9e6-002c-444d-88d8-1f666b55ce84","Type":"ContainerDied","Data":"84c230189ef32a08b296541172c2a963c832d4464df83e2c1e36361df9a6594f"}
Oct 31 16:35:53.203041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.202960    4417 scope.go:122] "RemoveContainer" containerID="5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"
Oct 31 16:35:53.203241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.203233    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"
Oct 31 16:35:53.210277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.210250    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-636a7f-openshift-default-dockercfg-lvhxg\""
Oct 31 16:35:53.224979 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.224949    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\") pod \"11daf9e6-002c-444d-88d8-1f666b55ce84\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") "
Oct 31 16:35:53.225082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.224995    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\") pod \"11daf9e6-002c-444d-88d8-1f666b55ce84\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") "
Oct 31 16:35:53.225125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225079    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\" (UniqueName: \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\") pod \"11daf9e6-002c-444d-88d8-1f666b55ce84\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") "
Oct 31 16:35:53.225161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225147    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") pod \"11daf9e6-002c-444d-88d8-1f666b55ce84\" (UID: \"11daf9e6-002c-444d-88d8-1f666b55ce84\") "
Oct 31 16:35:53.225662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225632    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.225802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225707    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.225866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225810    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.225924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225865    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.225924 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225900    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.226025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225923    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.226025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225954    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bj4\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.226025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.225982    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.226184 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.226040    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.226257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.226232    4417 scope.go:122] "RemoveContainer" containerID="5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"
Oct 31 16:35:53.226674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.226651    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca" pod "11daf9e6-002c-444d-88d8-1f666b55ce84" (UID: "11daf9e6-002c-444d-88d8-1f666b55ce84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:53.227672 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:53.226723    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484\": container with ID starting with 5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484 not found: ID does not exist" containerID="5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"
Oct 31 16:35:53.227672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.226760    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484"} err="failed to get container status \"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484\": rpc error: code = NotFound desc = could not find container \"5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484\": container with ID starting with 5d90926b8c8a7c7767121e3f49e6bfbb8bdfba8295399a23fbf50ed4a1d77484 not found: ID does not exist"
Oct 31 16:35:53.227672 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.226986    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11daf9e6-002c-444d-88d8-1f666b55ce84-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:53.233038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.229105    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls" pod "11daf9e6-002c-444d-88d8-1f666b55ce84" (UID: "11daf9e6-002c-444d-88d8-1f666b55ce84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:53.233038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.230971    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:35:53.233038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.231797    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89" pod "11daf9e6-002c-444d-88d8-1f666b55ce84" (UID: "11daf9e6-002c-444d-88d8-1f666b55ce84"). InnerVolumeSpecName "kube-api-access-qqn89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:53.234965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.234938    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs" pod "11daf9e6-002c-444d-88d8-1f666b55ce84" (UID: "11daf9e6-002c-444d-88d8-1f666b55ce84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:35:53.328524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328489    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328543    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328569    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328599    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bj4\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328663    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328717    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.328961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328945    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.328998    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329071    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329163    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:53.329235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329181    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqn89\" (UniqueName: \"kubernetes.io/projected/11daf9e6-002c-444d-88d8-1f666b55ce84-kube-api-access-qqn89\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:53.329235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329196    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11daf9e6-002c-444d-88d8-1f666b55ce84-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:53.329314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329290    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329467    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329485    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329663    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.329894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.329876    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.331271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.331242    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.331366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.331303    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.416641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.412571    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.416641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.415174    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bj4\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\") pod \"e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") " pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.486688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.474978    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:53.665469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.665438    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:35:53.672632 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:53.672590    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-hjt9x"]
Oct 31 16:35:54.084419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.019343    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:35:54.091637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.090806    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:54.091637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.090954    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:54.091637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.091061    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:35:54.212162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.212124    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerStarted","Data":"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"}
Oct 31 16:35:54.212290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.212186    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerStarted","Data":"43d74d43e5bb5f80603e6813b60ad3bcaab2a24ce9cc4ae9b3abad8f2daab4cf"}
Oct 31 16:35:54.320519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.320463    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podStartSLOduration=1.320444397 podStartE2EDuration="1.320444397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:54.318547184 +0000 UTC m=+1999.380122373" watchObservedRunningTime="2026-10-31 16:35:54.320444397 +0000 UTC m=+1999.382019586"
Oct 31 16:35:54.482692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:54.475752    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:55.051547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.051505    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11daf9e6-002c-444d-88d8-1f666b55ce84" path="/var/lib/kubelet/pods/11daf9e6-002c-444d-88d8-1f666b55ce84/volumes"
Oct 31 16:35:55.484075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.483986    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:55.818808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.818782    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:35:55.890530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890500    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890546    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890571    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890687    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890734    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890818    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.890933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890886    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.891009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890942    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.891073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891026    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\") pod \"52379bc0-1166-4cdb-840c-7d44348dee1a\" (UID: \"52379bc0-1166-4cdb-840c-7d44348dee1a\") "
Oct 31 16:35:55.891175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.890815    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.891238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891144    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.891419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891381    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.891550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891477    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.891627 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891557    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:35:55.891945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891925    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.892037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891951    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.892037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891970    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52379bc0-1166-4cdb-840c-7d44348dee1a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.892037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891985    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.892037 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.891999    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.893102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.893076    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:35:55.893271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.893255    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "kube-api-access-dn6ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:55.893352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.893300    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:35:55.893352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.893320    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token" pod "52379bc0-1166-4cdb-840c-7d44348dee1a" (UID: "52379bc0-1166-4cdb-840c-7d44348dee1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:35:55.993179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.993149    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.993179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.993176    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.993399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.993190    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6ps\" (UniqueName: \"kubernetes.io/projected/52379bc0-1166-4cdb-840c-7d44348dee1a-kube-api-access-dn6ps\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:55.993399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:55.993204    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52379bc0-1166-4cdb-840c-7d44348dee1a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:35:56.113467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.113437    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:35:56.113961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.113507    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:35:56.228685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.228659    4417 generic.go:356] "Generic (PLEG): container finished" podID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerID="a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22" exitCode=0
Oct 31 16:35:56.228886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.228701    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"
Oct 31 16:35:56.228886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.228768    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" event={"ID":"52379bc0-1166-4cdb-840c-7d44348dee1a","Type":"ContainerDied","Data":"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"}
Oct 31 16:35:56.228886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.228822    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2" event={"ID":"52379bc0-1166-4cdb-840c-7d44348dee1a","Type":"ContainerDied","Data":"6df62a53434b4674280da33a07612f70881962b01a56dc1938d6375e8e35511a"}
Oct 31 16:35:56.228886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.228850    4417 scope.go:122] "RemoveContainer" containerID="a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"
Oct 31 16:35:56.229111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.229040    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:56.230259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.230239    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:35:56.243565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.243548    4417 scope.go:122] "RemoveContainer" containerID="a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"
Oct 31 16:35:56.243916 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:35:56.243885    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22\": container with ID starting with a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22 not found: ID does not exist" containerID="a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"
Oct 31 16:35:56.243974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.243928    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22"} err="failed to get container status \"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22\": rpc error: code = NotFound desc = could not find container \"a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22\": container with ID starting with a83b535312adb5da9a9c60aeee3fcfeff1589393fbea998c1c3d4ebdbc926d22 not found: ID does not exist"
Oct 31 16:35:56.534539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.534450    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:35:56.538753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:56.538643    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-gfsl2"]
Oct 31 16:35:57.051425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:35:57.051390    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" path="/var/lib/kubelet/pods/52379bc0-1166-4cdb-840c-7d44348dee1a/volumes"
Oct 31 16:36:00.675807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.675768    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:36:00.676522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.676502    4417 cpu_manager.go:414] "containerMap: removing container" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy"
Oct 31 16:36:00.676606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.676523    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy"
Oct 31 16:36:00.676679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.676648    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="52379bc0-1166-4cdb-840c-7d44348dee1a" containerName="istio-proxy"
Oct 31 16:36:00.680157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.680135    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.731950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.731924    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:36:00.780639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.773797    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-636a7f\""
Oct 31 16:36:00.780639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.773994    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-636a7f-dockercfg-859pv\""
Oct 31 16:36:00.784666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.781130    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfplq\" (UniqueName: \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.784666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.782046    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.942019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.941934    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:36:00.967689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.942394    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfplq\" (UniqueName: \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.968133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.968103    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.969496 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.969461    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:00.973844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.973824    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:00.996116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:00.996078    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:36:01.018103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.018064    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfplq\" (UniqueName: \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\") pod \"payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:01.069926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.069893    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.070079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.069947    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcgs\" (UniqueName: \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.171596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.171560    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.171785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.171603    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcgs\" (UniqueName: \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.172305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.172279    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.224279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.224201    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcgs\" (UniqueName: \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\") pod \"payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.295989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.295953    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:01.315175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.315149    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:01.679443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.679408    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:36:01.719803 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:36:01.719763    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91f0df2_6f27_413f_ad30_ba80357cb953.slice/crio-a0b761eb58ab61638da3b8c2aef178cb096cbc88280069d8104da62249f63396 WatchSource:0}: Error finding container a0b761eb58ab61638da3b8c2aef178cb096cbc88280069d8104da62249f63396: Status 404 returned error can't find the container with id a0b761eb58ab61638da3b8c2aef178cb096cbc88280069d8104da62249f63396
Oct 31 16:36:01.729872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:01.729846    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:36:01.733767 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:36:01.733730    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ea2d49_7ec1_4778_9f81_c5793513f16b.slice/crio-a203eeb69c9a5b257e3b4c51495ead2cf3c82afd732cffd1132ef46262e88d47 WatchSource:0}: Error finding container a203eeb69c9a5b257e3b4c51495ead2cf3c82afd732cffd1132ef46262e88d47: Status 404 returned error can't find the container with id a203eeb69c9a5b257e3b4c51495ead2cf3c82afd732cffd1132ef46262e88d47
Oct 31 16:36:02.278324 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.278277    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" event={"ID":"b91f0df2-6f27-413f-ad30-ba80357cb953","Type":"ContainerStarted","Data":"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"}
Oct 31 16:36:02.278540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.278370    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:02.278540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.278404    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" event={"ID":"b91f0df2-6f27-413f-ad30-ba80357cb953","Type":"ContainerStarted","Data":"a0b761eb58ab61638da3b8c2aef178cb096cbc88280069d8104da62249f63396"}
Oct 31 16:36:02.280281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.280248    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" event={"ID":"13ea2d49-7ec1-4778-9f81-c5793513f16b","Type":"ContainerStarted","Data":"db0dde97385f4aa3e06bb669564831d6c1d8ad60a4e1b70744a0f3733911bb0b"}
Oct 31 16:36:02.280422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.280302    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" event={"ID":"13ea2d49-7ec1-4778-9f81-c5793513f16b","Type":"ContainerStarted","Data":"a203eeb69c9a5b257e3b4c51495ead2cf3c82afd732cffd1132ef46262e88d47"}
Oct 31 16:36:02.280422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.280380    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:02.384336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.384279    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" podStartSLOduration=2.384263519 podStartE2EDuration="2.384263519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:02.383059061 +0000 UTC m=+2007.444634375" watchObservedRunningTime="2026-10-31 16:36:02.384263519 +0000 UTC m=+2007.445838704"
Oct 31 16:36:02.491595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:02.490473    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" podStartSLOduration=2.490456119 podStartE2EDuration="2.490456119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:02.486956137 +0000 UTC m=+2007.548531323" watchObservedRunningTime="2026-10-31 16:36:02.490456119 +0000 UTC m=+2007.552031306"
Oct 31 16:36:02.949836 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:02.949797    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:36:07.621796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.620992    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:36:07.627585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.627539    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.651533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.651498    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.651739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.651656    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62xm\" (UniqueName: \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.667089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.667045    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:36:07.695638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.688127    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a1b30a15-dockercfg-tgkq8\""
Oct 31 16:36:07.695638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.688379    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a1b30a15\""
Oct 31 16:36:07.793642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.780866    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.793642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.781073    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62xm\" (UniqueName: \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.897473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.897344    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62xm\" (UniqueName: \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.907443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.907406    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.914703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.914670    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:36:07.919265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.919238    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:36:07.919397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.919343    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:07.949038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.949011    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:07.988289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.988257    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:07.988289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:07.988302    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsfz\" (UniqueName: \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.089537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.089501    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.089719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.089565    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsfz\" (UniqueName: \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.090664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.090601    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.142721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.142685    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsfz\" (UniqueName: \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\") pod \"payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") " pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.245862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.245763    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:08.336308 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.336262    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:36:08.456535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.456505    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" event={"ID":"d3399c25-2c2d-43e8-aba3-36b6350615ab","Type":"ContainerStarted","Data":"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"}
Oct 31 16:36:08.456683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.456562    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" event={"ID":"d3399c25-2c2d-43e8-aba3-36b6350615ab","Type":"ContainerStarted","Data":"b061d045043902bd8d96d8e080f5dad5f150c8c2ca6ed74ab812e818b62d53af"}
Oct 31 16:36:08.456683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.456601    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:08.563471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.563408    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" podStartSLOduration=1.563391516 podStartE2EDuration="1.563391516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:08.562426465 +0000 UTC m=+2013.624001650" watchObservedRunningTime="2026-10-31 16:36:08.563391516 +0000 UTC m=+2013.624966702"
Oct 31 16:36:08.620166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:08.620129    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:36:08.622328 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:36:08.622300    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b7bc81_0b35_46f1_aea3_1d8d417ee2ed.slice/crio-c9b5d77905dfb57540e66720e0e32cf81ff35a6fdf189b44d380e42d86831133 WatchSource:0}: Error finding container c9b5d77905dfb57540e66720e0e32cf81ff35a6fdf189b44d380e42d86831133: Status 404 returned error can't find the container with id c9b5d77905dfb57540e66720e0e32cf81ff35a6fdf189b44d380e42d86831133
Oct 31 16:36:09.467452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:09.467413    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" event={"ID":"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed","Type":"ContainerStarted","Data":"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"}
Oct 31 16:36:09.467653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:09.467473    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" event={"ID":"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed","Type":"ContainerStarted","Data":"c9b5d77905dfb57540e66720e0e32cf81ff35a6fdf189b44d380e42d86831133"}
Oct 31 16:36:09.467653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:09.467515    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:09.573379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:09.573321    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" podStartSLOduration=2.5733056149999998 podStartE2EDuration="2.573305615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:09.572384718 +0000 UTC m=+2014.633959903" watchObservedRunningTime="2026-10-31 16:36:09.573305615 +0000 UTC m=+2014.634880842"
Oct 31 16:36:13.290246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:13.290209    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:36:13.290692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:13.290292    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:36:14.457563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.457534    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:36:14.458074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.457883    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" podUID="ac029bba-1128-4098-8c80-268574190a1c" containerName="maas-api" containerID="cri-o://03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e" gracePeriod=30
Oct 31 16:36:14.810871 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.810836    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:36:14.951931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.925331    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") pod \"ac029bba-1128-4098-8c80-268574190a1c\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") "
Oct 31 16:36:14.951931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.925417    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\") pod \"ac029bba-1128-4098-8c80-268574190a1c\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") "
Oct 31 16:36:14.951931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.925471    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\") pod \"ac029bba-1128-4098-8c80-268574190a1c\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") "
Oct 31 16:36:14.951931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.925804    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\" (UniqueName: \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\") pod \"ac029bba-1128-4098-8c80-268574190a1c\" (UID: \"ac029bba-1128-4098-8c80-268574190a1c\") "
Oct 31 16:36:14.951931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.951875    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca" pod "ac029bba-1128-4098-8c80-268574190a1c" (UID: "ac029bba-1128-4098-8c80-268574190a1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:14.975668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.964965    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x" pod "ac029bba-1128-4098-8c80-268574190a1c" (UID: "ac029bba-1128-4098-8c80-268574190a1c"). InnerVolumeSpecName "kube-api-access-4298x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:14.975668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.969514    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls" pod "ac029bba-1128-4098-8c80-268574190a1c" (UID: "ac029bba-1128-4098-8c80-268574190a1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:14.986812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:14.978848    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs" pod "ac029bba-1128-4098-8c80-268574190a1c" (UID: "ac029bba-1128-4098-8c80-268574190a1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:15.030545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.030450    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac029bba-1128-4098-8c80-268574190a1c-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:15.030545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.030488    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4298x\" (UniqueName: \"kubernetes.io/projected/ac029bba-1128-4098-8c80-268574190a1c-kube-api-access-4298x\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:15.030545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.030504    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:15.030545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.030518    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac029bba-1128-4098-8c80-268574190a1c-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:15.512756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.512725    4417 generic.go:356] "Generic (PLEG): container finished" podID="ac029bba-1128-4098-8c80-268574190a1c" containerID="03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e" exitCode=0
Oct 31 16:36:15.513216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.512788    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"
Oct 31 16:36:15.513216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.512820    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" event={"ID":"ac029bba-1128-4098-8c80-268574190a1c","Type":"ContainerDied","Data":"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"}
Oct 31 16:36:15.513216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.512882    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv" event={"ID":"ac029bba-1128-4098-8c80-268574190a1c","Type":"ContainerDied","Data":"190304d229203c7f8e3abbc34326ca1dadcbe2d07ba7725e48754b9f70ab0656"}
Oct 31 16:36:15.513216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.512910    4417 scope.go:122] "RemoveContainer" containerID="03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"
Oct 31 16:36:15.526491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.525256    4417 scope.go:122] "RemoveContainer" containerID="03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"
Oct 31 16:36:15.526491 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:15.525605    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e\": container with ID starting with 03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e not found: ID does not exist" containerID="03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"
Oct 31 16:36:15.526491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.525650    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e"} err="failed to get container status \"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e\": rpc error: code = NotFound desc = could not find container \"03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e\": container with ID starting with 03e4f8ae0f467dfcd57f557cf2892c8c41f23ed45c4e909fabd0062c0481a04e not found: ID does not exist"
Oct 31 16:36:15.683641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.675080    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:36:15.736645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:15.734347    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a2ae37-99959b4cf-gwpdv"]
Oct 31 16:36:17.052201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:17.052120    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac029bba-1128-4098-8c80-268574190a1c" path="/var/lib/kubelet/pods/ac029bba-1128-4098-8c80-268574190a1c/volumes"
Oct 31 16:36:19.471075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:19.471046    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:36:20.478935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:20.478900    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:36:30.726067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:30.726032    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:36:30.726527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:30.726275    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" podUID="0d968dd0-128a-40a9-a180-23b9d1575526" containerName="maas-api" containerID="cri-o://7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55" gracePeriod=30
Oct 31 16:36:31.092406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.092379    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:36:31.097951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.097928    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\") pod \"0d968dd0-128a-40a9-a180-23b9d1575526\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") "
Oct 31 16:36:31.098071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.097988    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\") pod \"0d968dd0-128a-40a9-a180-23b9d1575526\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") "
Oct 31 16:36:31.098120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.098097    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\" (UniqueName: \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\") pod \"0d968dd0-128a-40a9-a180-23b9d1575526\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") "
Oct 31 16:36:31.098162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.098122    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\") pod \"0d968dd0-128a-40a9-a180-23b9d1575526\" (UID: \"0d968dd0-128a-40a9-a180-23b9d1575526\") "
Oct 31 16:36:31.098274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.098252    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca" pod "0d968dd0-128a-40a9-a180-23b9d1575526" (UID: "0d968dd0-128a-40a9-a180-23b9d1575526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:31.098544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.098525    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d968dd0-128a-40a9-a180-23b9d1575526-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:31.100207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.100186    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8" pod "0d968dd0-128a-40a9-a180-23b9d1575526" (UID: "0d968dd0-128a-40a9-a180-23b9d1575526"). InnerVolumeSpecName "kube-api-access-ngzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:31.100482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.100459    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls" pod "0d968dd0-128a-40a9-a180-23b9d1575526" (UID: "0d968dd0-128a-40a9-a180-23b9d1575526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:31.100522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.100475    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs" pod "0d968dd0-128a-40a9-a180-23b9d1575526" (UID: "0d968dd0-128a-40a9-a180-23b9d1575526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:31.203639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.203145    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:31.203639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.203185    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzr8\" (UniqueName: \"kubernetes.io/projected/0d968dd0-128a-40a9-a180-23b9d1575526-kube-api-access-ngzr8\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:31.203639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.203201    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d968dd0-128a-40a9-a180-23b9d1575526-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:31.628417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.628384    4417 generic.go:356] "Generic (PLEG): container finished" podID="0d968dd0-128a-40a9-a180-23b9d1575526" containerID="7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55" exitCode=0
Oct 31 16:36:31.628607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.628433    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"
Oct 31 16:36:31.628607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.628463    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" event={"ID":"0d968dd0-128a-40a9-a180-23b9d1575526","Type":"ContainerDied","Data":"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"}
Oct 31 16:36:31.628607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.628518    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p" event={"ID":"0d968dd0-128a-40a9-a180-23b9d1575526","Type":"ContainerDied","Data":"e37ca1f918c5ff9ac95c23686504e108b3b92adf40db1b7d1676abff334d78d4"}
Oct 31 16:36:31.628607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.628544    4417 scope.go:122] "RemoveContainer" containerID="7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"
Oct 31 16:36:31.640085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.640068    4417 scope.go:122] "RemoveContainer" containerID="7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"
Oct 31 16:36:31.640399 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:31.640380    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55\": container with ID starting with 7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55 not found: ID does not exist" containerID="7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"
Oct 31 16:36:31.640459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.640405    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55"} err="failed to get container status \"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55\": rpc error: code = NotFound desc = could not find container \"7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55\": container with ID starting with 7a15c4ad8eb298201884f141983c7cd5b3b4540a9c02f293ab0ab75cecac6a55 not found: ID does not exist"
Oct 31 16:36:31.825640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.824667    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:36:31.839575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.839532    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6b1cca-7954c46d84-rjm6p"]
Oct 31 16:36:31.901356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.901249    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:36:31.901772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:31.901741    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" podUID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" containerName="maas-api" containerID="cri-o://dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63" gracePeriod=30
Oct 31 16:36:32.251633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.251587    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:36:32.377611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.376153    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\") pod \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") "
Oct 31 16:36:32.377611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.376431    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\" (UniqueName: \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\") pod \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") "
Oct 31 16:36:32.377611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.376502    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\") pod \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") "
Oct 31 16:36:32.377611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.376572    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") pod \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\" (UID: \"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a\") "
Oct 31 16:36:32.412208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.412169    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca" pod "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" (UID: "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:32.439651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.437012    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls" pod "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" (UID: "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:32.439651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.437212    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp" pod "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" (UID: "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a"). InnerVolumeSpecName "kube-api-access-hvgpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:32.439651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.437939    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs" pod "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" (UID: "bdfc31c6-2c77-4d27-b70c-d20ea6c4946a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:36:32.480697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.478839    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvgpp\" (UniqueName: \"kubernetes.io/projected/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-kube-api-access-hvgpp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:32.480697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.478876    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:32.480697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.478899    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:32.480697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.478912    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:32.636428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.636397    4417 generic.go:356] "Generic (PLEG): container finished" podID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" containerID="dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63" exitCode=0
Oct 31 16:36:32.636580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.636445    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"
Oct 31 16:36:32.636580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.636483    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" event={"ID":"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a","Type":"ContainerDied","Data":"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"}
Oct 31 16:36:32.636580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.636537    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h" event={"ID":"bdfc31c6-2c77-4d27-b70c-d20ea6c4946a","Type":"ContainerDied","Data":"e2423b02cce2b4e88f4832d9fe2d39e994ae92840b423b7d43170511a16ff444"}
Oct 31 16:36:32.636580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.636558    4417 scope.go:122] "RemoveContainer" containerID="dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"
Oct 31 16:36:32.646576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.646560    4417 scope.go:122] "RemoveContainer" containerID="dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"
Oct 31 16:36:32.646872 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:32.646853    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63\": container with ID starting with dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63 not found: ID does not exist" containerID="dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"
Oct 31 16:36:32.646923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.646879    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63"} err="failed to get container status \"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63\": rpc error: code = NotFound desc = could not find container \"dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63\": container with ID starting with dba0f3781a10193b28b2c333cba6b47e153ac9010f9a0f9199a25b8595f33a63 not found: ID does not exist"
Oct 31 16:36:32.793488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.793454    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:36:32.835438 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:32.835411    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-822dd4-75cf5987bd-d7r4h"]
Oct 31 16:36:33.049700 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:33.049566    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:36:33.057926 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:33.057894    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d968dd0-128a-40a9-a180-23b9d1575526" path="/var/lib/kubelet/pods/0d968dd0-128a-40a9-a180-23b9d1575526/volumes"
Oct 31 16:36:33.058815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:33.058782    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" path="/var/lib/kubelet/pods/bdfc31c6-2c77-4d27-b70c-d20ea6c4946a/volumes"
Oct 31 16:36:39.384349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:39.384316    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:36:39.385103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:39.384511    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" containerID="cri-o://cc14e1c21871cab1051e9461458c4413325bbfda18a1e95bd0ce67c2ea5a3a36" gracePeriod=30
Oct 31 16:36:40.912375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:40.912340    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:36:40.912881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:40.912595    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" containerID="cri-o://ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983" gracePeriod=30
Oct 31 16:36:41.227032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:41.226937    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:36:41.227262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:41.227237    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:36:42.929104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:42.929054    4417 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:36:42.929464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:42.929127    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:36:44.727821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.727786    4417 generic.go:356] "Generic (PLEG): container finished" podID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerID="cc14e1c21871cab1051e9461458c4413325bbfda18a1e95bd0ce67c2ea5a3a36" exitCode=0
Oct 31 16:36:44.728253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.727863    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerDied","Data":"cc14e1c21871cab1051e9461458c4413325bbfda18a1e95bd0ce67c2ea5a3a36"}
Oct 31 16:36:44.728253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.727940    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh" event={"ID":"e6a880d6-5914-4b05-9706-d4510a9a34ab","Type":"ContainerDied","Data":"ef6f15402a6da9673fd006836bbd695471b9b4a91639840a705d3b886c0f0ea6"}
Oct 31 16:36:44.728253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.727971    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef6f15402a6da9673fd006836bbd695471b9b4a91639840a705d3b886c0f0ea6"
Oct 31 16:36:44.728253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.727993    4417 scope.go:122] "RemoveContainer" containerID="c64f3b38258e4595a8f630e3d86197e7080f3fe4c6b27744a089ca20f5dc8a0b"
Oct 31 16:36:44.734019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.733992    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:36:44.851482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851449    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851510    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851600    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851661    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851741    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851771    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851834    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.851937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851831    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:44.851937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851928    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.852195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.851973    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\") pod \"e6a880d6-5914-4b05-9706-d4510a9a34ab\" (UID: \"e6a880d6-5914-4b05-9706-d4510a9a34ab\") "
Oct 31 16:36:44.852195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852022    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:44.852195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852139    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:44.852370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852203    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:44.852439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852383    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:44.852812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852776    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.852812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852804    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.852996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852822    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a880d6-5914-4b05-9706-d4510a9a34ab-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.852996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852837    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.852996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.852851    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.854048 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.854022    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:44.854166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.854097    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:36:44.854258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.854241    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:44.854299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.854279    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4" pod "e6a880d6-5914-4b05-9706-d4510a9a34ab" (UID: "e6a880d6-5914-4b05-9706-d4510a9a34ab"). InnerVolumeSpecName "kube-api-access-r2wz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:44.954137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.954052    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.954137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.954081    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.954137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.954090    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a880d6-5914-4b05-9706-d4510a9a34ab-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:44.954137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:44.954099    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wz4\" (UniqueName: \"kubernetes.io/projected/e6a880d6-5914-4b05-9706-d4510a9a34ab-kube-api-access-r2wz4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:45.735339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:45.735310    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"
Oct 31 16:36:46.006008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.005912    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:36:46.007199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.007163    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6b1cca-openshift-default-644dc8c7bc-jsjzh"]
Oct 31 16:36:46.203633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.203601    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:36:46.271392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271296    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271381    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271407    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271426    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271456    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271492    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271549    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271610    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:46.271961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271733    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271792    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\") pod \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\" (UID: \"241b81b1-5aa3-4bd4-8e17-f857aceb4a87\") "
Oct 31 16:36:46.271961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271882    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:46.271961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.271933    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:46.272174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272136    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:46.272559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272519    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.272559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272549    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.272849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272565    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.272849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272580    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.272849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.272728    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:46.273906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.273877    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:36:46.274064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.274042    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "kube-api-access-c6nz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:46.274226 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.274210    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:46.274281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.274260    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo" pod "241b81b1-5aa3-4bd4-8e17-f857aceb4a87" (UID: "241b81b1-5aa3-4bd4-8e17-f857aceb4a87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:36:46.374071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.374042    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nz4\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-kube-api-access-c6nz4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.374071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.374070    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.374275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.374080    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.374275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.374089    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.374275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.374097    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/241b81b1-5aa3-4bd4-8e17-f857aceb4a87-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:46.743231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.743196    4417 generic.go:356] "Generic (PLEG): container finished" podID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerID="ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983" exitCode=0
Oct 31 16:36:46.743678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.743248    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"
Oct 31 16:36:46.743678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.743282    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerDied","Data":"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"}
Oct 31 16:36:46.743678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.743339    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq" event={"ID":"241b81b1-5aa3-4bd4-8e17-f857aceb4a87","Type":"ContainerDied","Data":"c1e0d426b9bd9c5baed0401be4511e3101af85149c392160450f21cd8d28297f"}
Oct 31 16:36:46.743678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.743364    4417 scope.go:122] "RemoveContainer" containerID="ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"
Oct 31 16:36:46.755627 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.755600    4417 scope.go:122] "RemoveContainer" containerID="7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"
Oct 31 16:36:46.764737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.764716    4417 scope.go:122] "RemoveContainer" containerID="ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"
Oct 31 16:36:46.765071 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:46.765046    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983\": container with ID starting with ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983 not found: ID does not exist" containerID="ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"
Oct 31 16:36:46.765181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.765078    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983"} err="failed to get container status \"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983\": rpc error: code = NotFound desc = could not find container \"ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983\": container with ID starting with ba046b3fba177c80abdd8f50d023cc4ac4726c1ebc9337ef7217124d1c14e983 not found: ID does not exist"
Oct 31 16:36:46.765181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.765096    4417 scope.go:122] "RemoveContainer" containerID="7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"
Oct 31 16:36:46.765384 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:46.765365    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8\": container with ID starting with 7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8 not found: ID does not exist" containerID="7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"
Oct 31 16:36:46.765435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.765390    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8"} err="failed to get container status \"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8\": rpc error: code = NotFound desc = could not find container \"7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8\": container with ID starting with 7a8a84c3538d882fd98fee2ee043baaf07cac40e2e5d0490220411ebf4106bf8 not found: ID does not exist"
Oct 31 16:36:46.896483 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.896454    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:36:46.942147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:46.942111    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-822dd4-openshift-default-854cd7994c-hj9qq"]
Oct 31 16:36:47.053010 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:47.052971    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" path="/var/lib/kubelet/pods/241b81b1-5aa3-4bd4-8e17-f857aceb4a87/volumes"
Oct 31 16:36:47.053720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:47.053700    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" path="/var/lib/kubelet/pods/e6a880d6-5914-4b05-9706-d4510a9a34ab/volumes"
Oct 31 16:36:51.621152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:51.621115    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:36:51.720144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:51.720114    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:36:53.928907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:53.928874    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:36:54.026575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:54.026543    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:36:56.621464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:56.621429    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" podUID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" containerName="payload-processing" containerID="cri-o://7da883ce1d123fd36290de2ab54989d4c2c1d8e3b0b772ac0aac7bbe1e154b2c" gracePeriod=25
Oct 31 16:36:56.721214 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:56.721180    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" podUID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" containerName="payload-pre-processing" containerID="cri-o://defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0" gracePeriod=25
Oct 31 16:36:56.825154 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:56.825120    4417 generic.go:356] "Generic (PLEG): container finished" podID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" containerID="7da883ce1d123fd36290de2ab54989d4c2c1d8e3b0b772ac0aac7bbe1e154b2c" exitCode=0
Oct 31 16:36:56.825337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:56.825229    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" event={"ID":"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8","Type":"ContainerDied","Data":"7da883ce1d123fd36290de2ab54989d4c2c1d8e3b0b772ac0aac7bbe1e154b2c"}
Oct 31 16:36:56.909523 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:56.909500    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:36:57.009193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.009169    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:36:57.059542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.059517    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\" (UniqueName: \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\") pod \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") "
Oct 31 16:36:57.060116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.060096    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\") pod \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\" (UID: \"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8\") "
Oct 31 16:36:57.060825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.060671    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume" pod "6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" (UID: "6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:57.060974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.060957    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\" (UniqueName: \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\") pod \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") "
Oct 31 16:36:57.061157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.061141    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\") pod \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\" (UID: \"9407fa21-addb-4e36-aaf9-eb984c4f6f5a\") "
Oct 31 16:36:57.061907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.061795    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume" pod "9407fa21-addb-4e36-aaf9-eb984c4f6f5a" (UID: "9407fa21-addb-4e36-aaf9-eb984c4f6f5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:57.062488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.062467    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:57.062585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.062491    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:57.062678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.062601    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz" pod "6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" (UID: "6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8"). InnerVolumeSpecName "kube-api-access-6t8rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:57.063406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.063378    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p" pod "9407fa21-addb-4e36-aaf9-eb984c4f6f5a" (UID: "9407fa21-addb-4e36-aaf9-eb984c4f6f5a"). InnerVolumeSpecName "kube-api-access-6hj7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:57.163600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.163515    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8rz\" (UniqueName: \"kubernetes.io/projected/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8-kube-api-access-6t8rz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:57.163600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.163548    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hj7p\" (UniqueName: \"kubernetes.io/projected/9407fa21-addb-4e36-aaf9-eb984c4f6f5a-kube-api-access-6hj7p\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:57.833255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.833221    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"
Oct 31 16:36:57.833721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.833267    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8" event={"ID":"6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8","Type":"ContainerDied","Data":"8949393c7ae98a0125470e73176b26a5447fdc052fa8c0cf215c65a90497bc2d"}
Oct 31 16:36:57.833721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.833326    4417 scope.go:122] "RemoveContainer" containerID="7da883ce1d123fd36290de2ab54989d4c2c1d8e3b0b772ac0aac7bbe1e154b2c"
Oct 31 16:36:57.834805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.834784    4417 generic.go:356] "Generic (PLEG): container finished" podID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" containerID="defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0" exitCode=0
Oct 31 16:36:57.834895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.834835    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"
Oct 31 16:36:57.834949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.834890    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" event={"ID":"9407fa21-addb-4e36-aaf9-eb984c4f6f5a","Type":"ContainerDied","Data":"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"}
Oct 31 16:36:57.834949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.834923    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj" event={"ID":"9407fa21-addb-4e36-aaf9-eb984c4f6f5a","Type":"ContainerDied","Data":"3f0cdbe5d96c8e15c9fa49017e8a3500a3d6d71476b8886c79b6bcf04d630cba"}
Oct 31 16:36:57.847753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.847735    4417 scope.go:122] "RemoveContainer" containerID="defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"
Oct 31 16:36:57.859981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.859960    4417 scope.go:122] "RemoveContainer" containerID="defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"
Oct 31 16:36:57.860342 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:57.860314    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0\": container with ID starting with defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0 not found: ID does not exist" containerID="defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"
Oct 31 16:36:57.860418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.860354    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0"} err="failed to get container status \"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0\": rpc error: code = NotFound desc = could not find container \"defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0\": container with ID starting with defcfc306eabf6dda57a427ca021763ea3cdf10a6345183ae4991de9831ffff0 not found: ID does not exist"
Oct 31 16:36:57.989986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.989907    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:36:57.994445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:57.994418    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-487213-7795765697-cdgg8"]
Oct 31 16:36:58.090796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:58.090759    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:36:58.096817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:58.096790    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-487213-869b67d7f9-ddqrj"]
Oct 31 16:36:58.930218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:58.930149    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" podUID="4c419478-af60-4db4-865e-b644f4b9164f" containerName="payload-processing" containerID="cri-o://b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361" gracePeriod=25
Oct 31 16:36:59.027629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.027563    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" podUID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" containerName="payload-pre-processing" containerID="cri-o://c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee" gracePeriod=25
Oct 31 16:36:59.055056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.055024    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" path="/var/lib/kubelet/pods/6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8/volumes"
Oct 31 16:36:59.055599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.055583    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" path="/var/lib/kubelet/pods/9407fa21-addb-4e36-aaf9-eb984c4f6f5a/volumes"
Oct 31 16:36:59.224304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.222251    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:36:59.288165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.288131    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\" (UniqueName: \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\") pod \"4c419478-af60-4db4-865e-b644f4b9164f\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") "
Oct 31 16:36:59.288325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.288228    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\") pod \"4c419478-af60-4db4-865e-b644f4b9164f\" (UID: \"4c419478-af60-4db4-865e-b644f4b9164f\") "
Oct 31 16:36:59.288648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.288593    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume" pod "4c419478-af60-4db4-865e-b644f4b9164f" (UID: "4c419478-af60-4db4-865e-b644f4b9164f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:59.288870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.288854    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c419478-af60-4db4-865e-b644f4b9164f-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:59.290267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.290238    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl" pod "4c419478-af60-4db4-865e-b644f4b9164f" (UID: "4c419478-af60-4db4-865e-b644f4b9164f"). InnerVolumeSpecName "kube-api-access-fk7kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:59.311988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.311967    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:36:59.390266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.390238    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\") pod \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") "
Oct 31 16:36:59.390266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.390270    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\" (UniqueName: \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\") pod \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\" (UID: \"4e4d932e-cdd9-4e04-ab7f-b849d8f55120\") "
Oct 31 16:36:59.390613 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.390590    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume" pod "4e4d932e-cdd9-4e04-ab7f-b849d8f55120" (UID: "4e4d932e-cdd9-4e04-ab7f-b849d8f55120"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:36:59.390849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.390832    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7kl\" (UniqueName: \"kubernetes.io/projected/4c419478-af60-4db4-865e-b644f4b9164f-kube-api-access-fk7kl\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:59.390900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.390852    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:59.392317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.392299    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm" pod "4e4d932e-cdd9-4e04-ab7f-b849d8f55120" (UID: "4e4d932e-cdd9-4e04-ab7f-b849d8f55120"). InnerVolumeSpecName "kube-api-access-gdnsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:36:59.492248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.492178    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnsm\" (UniqueName: \"kubernetes.io/projected/4e4d932e-cdd9-4e04-ab7f-b849d8f55120-kube-api-access-gdnsm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:36:59.851273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.851238    4417 generic.go:356] "Generic (PLEG): container finished" podID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" containerID="c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee" exitCode=0
Oct 31 16:36:59.851429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.851286    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"
Oct 31 16:36:59.851429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.851321    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" event={"ID":"4e4d932e-cdd9-4e04-ab7f-b849d8f55120","Type":"ContainerDied","Data":"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"}
Oct 31 16:36:59.851429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.851378    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm" event={"ID":"4e4d932e-cdd9-4e04-ab7f-b849d8f55120","Type":"ContainerDied","Data":"45f56e8aee0198dfe938bee0dae8ffb6b8c2195859e245001954b2e0f1944717"}
Oct 31 16:36:59.851429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.851405    4417 scope.go:122] "RemoveContainer" containerID="c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"
Oct 31 16:36:59.852995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.852972    4417 generic.go:356] "Generic (PLEG): container finished" podID="4c419478-af60-4db4-865e-b644f4b9164f" containerID="b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361" exitCode=0
Oct 31 16:36:59.853117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.853069    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"
Oct 31 16:36:59.853117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.853079    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" event={"ID":"4c419478-af60-4db4-865e-b644f4b9164f","Type":"ContainerDied","Data":"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"}
Oct 31 16:36:59.853242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.853120    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk" event={"ID":"4c419478-af60-4db4-865e-b644f4b9164f","Type":"ContainerDied","Data":"d4274e5c4fe36fe60d18ee370f5096dcf5fe1b2e5c7b2dfb1d25621cd6390a48"}
Oct 31 16:36:59.862883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.862860    4417 scope.go:122] "RemoveContainer" containerID="c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"
Oct 31 16:36:59.863247 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:59.863138    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee\": container with ID starting with c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee not found: ID does not exist" containerID="c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"
Oct 31 16:36:59.863247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.863165    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee"} err="failed to get container status \"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee\": rpc error: code = NotFound desc = could not find container \"c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee\": container with ID starting with c5e41de7a72323eaf2c21269a64fe140483a713cfbb77e958412ce789a1751ee not found: ID does not exist"
Oct 31 16:36:59.863247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.863180    4417 scope.go:122] "RemoveContainer" containerID="b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"
Oct 31 16:36:59.872347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.872330    4417 scope.go:122] "RemoveContainer" containerID="b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"
Oct 31 16:36:59.872595 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:36:59.872576    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361\": container with ID starting with b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361 not found: ID does not exist" containerID="b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"
Oct 31 16:36:59.872668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:36:59.872602    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361"} err="failed to get container status \"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361\": rpc error: code = NotFound desc = could not find container \"b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361\": container with ID starting with b95c0b5fa3aae9d3b91284602edab023057b79c2acb4594a2caa5e3d3e178361 not found: ID does not exist"
Oct 31 16:37:00.005833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:00.005760    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:37:00.051289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:00.051260    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d0755f-dc9b99cbc-rt7tm"]
Oct 31 16:37:00.154826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:00.154746    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:37:00.160775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:00.160750    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d0755f-d59ff87cc-phddk"]
Oct 31 16:37:01.050811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:01.050783    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c419478-af60-4db4-865e-b644f4b9164f" path="/var/lib/kubelet/pods/4c419478-af60-4db4-865e-b644f4b9164f/volumes"
Oct 31 16:37:01.051231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:01.051217    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" path="/var/lib/kubelet/pods/4e4d932e-cdd9-4e04-ab7f-b849d8f55120/volumes"
Oct 31 16:37:02.945070 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:02.945028    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:37:04.659767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.659193    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660142    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4c419478-af60-4db4-865e-b644f4b9164f" containerName="payload-processing"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660161    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c419478-af60-4db4-865e-b644f4b9164f" containerName="payload-processing"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660179    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660186    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660203    4417 cpu_manager.go:414] "containerMap: removing container" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660211    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660225    4417 cpu_manager.go:414] "containerMap: removing container" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660233    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660246    4417 cpu_manager.go:414] "containerMap: removing container" podUID="ac029bba-1128-4098-8c80-268574190a1c" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660253    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac029bba-1128-4098-8c80-268574190a1c" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660270    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" containerName="payload-pre-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660277    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" containerName="payload-pre-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660289    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" containerName="payload-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660297    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" containerName="payload-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660325    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" containerName="payload-pre-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660333    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" containerName="payload-pre-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660355    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660362    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660377    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0d968dd0-128a-40a9-a180-23b9d1575526" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660384    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d968dd0-128a-40a9-a180-23b9d1575526" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660397    4417 cpu_manager.go:414] "containerMap: removing container" podUID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660404    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" containerName="maas-api"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660562    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c419478-af60-4db4-865e-b644f4b9164f" containerName="payload-processing"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660576    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660586    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660597    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="241b81b1-5aa3-4bd4-8e17-f857aceb4a87" containerName="istio-proxy"
Oct 31 16:37:04.660603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660605    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac029bba-1128-4098-8c80-268574190a1c" containerName="maas-api"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660630    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9407fa21-addb-4e36-aaf9-eb984c4f6f5a" containerName="payload-pre-processing"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660640    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abbdee1-7a39-4aaa-b7b6-68b0e322e5f8" containerName="payload-processing"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660652    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4d932e-cdd9-4e04-ab7f-b849d8f55120" containerName="payload-pre-processing"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660662    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a880d6-5914-4b05-9706-d4510a9a34ab" containerName="istio-proxy"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660673    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d968dd0-128a-40a9-a180-23b9d1575526" containerName="maas-api"
Oct 31 16:37:04.661536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.660684    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdfc31c6-2c77-4d27-b70c-d20ea6c4946a" containerName="maas-api"
Oct 31 16:37:04.666302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.666275    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.715336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.715292    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-05db63-serving-cert\""
Oct 31 16:37:04.718953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.718918    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:37:04.762499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.762458    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.762730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.762660    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.762824 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.762786    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9zb\" (UniqueName: \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.762921 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.762903    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.864639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.864580    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.864834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.864682    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9zb\" (UniqueName: \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.864834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.864759    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.864834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.864820    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.864994 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:04.864758    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-serving-cert: secret "maas-api-e2e-inf-a-05db63-serving-cert" not found
Oct 31 16:37:04.865488 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:04.865450    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls podName:2e0be038-7784-429a-95cb-fbdbcf236917 nodeName:}" failed. No retries permitted until 2026-10-31 16:37:05.364895706 +0000 UTC m=+2070.426470878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls") pod "maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" (UID: "2e0be038-7784-429a-95cb-fbdbcf236917") : secret "maas-api-e2e-inf-a-05db63-serving-cert" not found
Oct 31 16:37:04.865610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.865581    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.867325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.867300    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:04.917992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:04.917921    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9zb\" (UniqueName: \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:05.373219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:05.373186    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:05.375812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:05.375781    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") pod \"maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:05.582570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:05.582534    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:05.962733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:05.962700    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:37:05.975399 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:05.975373    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:37:06.905222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:06.905192    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" event={"ID":"2e0be038-7784-429a-95cb-fbdbcf236917","Type":"ContainerStarted","Data":"0c8339edef6b24ee4cdcf36d6515e4f17dedd2135d3bf2985b454972d84e227a"}
Oct 31 16:37:07.913075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:07.913034    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" event={"ID":"2e0be038-7784-429a-95cb-fbdbcf236917","Type":"ContainerStarted","Data":"f33429eec0b7b29aec64c9c24b46fa8ca809b27447ce04fdcd93895c4c652230"}
Oct 31 16:37:07.913473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:07.913102    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:08.021490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:08.021439    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" podStartSLOduration=3.10371102 podStartE2EDuration="4.021424173s" totalImagesPullingTime="917.713153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:08.02022123 +0000 UTC m=+2073.081796417" watchObservedRunningTime="2026-10-31 16:37:08.021424173 +0000 UTC m=+2073.082999358"
Oct 31 16:37:13.938527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:13.938498    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:37:15.636090 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.636056    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:37:15.640891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.640871    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.661915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.661887    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.662055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.661921    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.662055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.661949    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmb6\" (UniqueName: \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.662134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.662055    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.689009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.688982    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a1b30a15-serving-cert\""
Oct 31 16:37:15.738709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.738681    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:37:15.762860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.762832    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.763032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.762866    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.763032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.762897    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmb6\" (UniqueName: \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.763032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.762939    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.763032 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:15.763016    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-serving-cert: secret "maas-api-e2e-mt-a1b30a15-serving-cert" not found
Oct 31 16:37:15.763253 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:15.763098    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls podName:a5b53163-092a-440a-ae42-669f85d5ce5c nodeName:}" failed. No retries permitted until 2026-10-31 16:37:16.263076884 +0000 UTC m=+2081.324652048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls") pod "maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" (UID: "a5b53163-092a-440a-ae42-669f85d5ce5c") : secret "maas-api-e2e-mt-a1b30a15-serving-cert" not found
Oct 31 16:37:15.763647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.763610    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.765602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.765579    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:15.816549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:15.816514    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmb6\" (UniqueName: \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:16.268779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:16.268745    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:16.271358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:16.271329    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") pod \"maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:16.554711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:16.554672    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:16.924068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:16.924035    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:37:16.925478 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:37:16.925454    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b53163_092a_440a_ae42_669f85d5ce5c.slice/crio-b55ddd1646f70fe42a8b2059ffc1b57b6c61130d4c69109b144ee29d14c5893c WatchSource:0}: Error finding container b55ddd1646f70fe42a8b2059ffc1b57b6c61130d4c69109b144ee29d14c5893c: Status 404 returned error can't find the container with id b55ddd1646f70fe42a8b2059ffc1b57b6c61130d4c69109b144ee29d14c5893c
Oct 31 16:37:17.001252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:17.001219    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" event={"ID":"a5b53163-092a-440a-ae42-669f85d5ce5c","Type":"ContainerStarted","Data":"b55ddd1646f70fe42a8b2059ffc1b57b6c61130d4c69109b144ee29d14c5893c"}
Oct 31 16:37:18.010339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:18.010300    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" event={"ID":"a5b53163-092a-440a-ae42-669f85d5ce5c","Type":"ContainerStarted","Data":"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"}
Oct 31 16:37:18.010761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:18.010371    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:18.117773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:18.117663    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" podStartSLOduration=2.254051218 podStartE2EDuration="3.117648191s" totalImagesPullingTime="863.596973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:18.116009706 +0000 UTC m=+2083.177584891" watchObservedRunningTime="2026-10-31 16:37:18.117648191 +0000 UTC m=+2083.179223377"
Oct 31 16:37:24.022468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:24.022438    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:37:26.111692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.111655    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:37:26.116518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.116496    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.164699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.164668    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-636a7f-serving-cert\""
Oct 31 16:37:26.216337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.216300    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:37:26.223666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.223601    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.223947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.223919    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtphz\" (UniqueName: \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.224074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.224023    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.224142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.224096    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.326768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.326727    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtphz\" (UniqueName: \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.326974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.326839    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.326974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.326919    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.327086 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:26.326973    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-serving-cert: secret "maas-api-e2e-inf-a-636a7f-serving-cert" not found
Oct 31 16:37:26.327086 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:26.327057    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls podName:22cd68ee-011b-4fda-b725-58000a5b6b23 nodeName:}" failed. No retries permitted until 2026-10-31 16:37:26.827037009 +0000 UTC m=+2091.888612190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls") pod "maas-api-e2e-inf-a-636a7f-759b587b67-kph65" (UID: "22cd68ee-011b-4fda-b725-58000a5b6b23") : secret "maas-api-e2e-inf-a-636a7f-serving-cert" not found
Oct 31 16:37:26.327206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.327115    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.327873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.327851    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.329415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.329396    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.425482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.425218    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtphz\" (UniqueName: \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.836291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.836248    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:26.838943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:26.838907    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") pod \"maas-api-e2e-inf-a-636a7f-759b587b67-kph65\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:27.031168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:27.031136    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:27.404393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:27.404363    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:37:27.407240 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:37:27.407211    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cd68ee_011b_4fda_b725_58000a5b6b23.slice/crio-691ba8fb8930b41f7f59773dcf0d4779da9bb3d3ea1a7d48d2347d024994ea00 WatchSource:0}: Error finding container 691ba8fb8930b41f7f59773dcf0d4779da9bb3d3ea1a7d48d2347d024994ea00: Status 404 returned error can't find the container with id 691ba8fb8930b41f7f59773dcf0d4779da9bb3d3ea1a7d48d2347d024994ea00
Oct 31 16:37:28.114109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:28.114071    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" event={"ID":"22cd68ee-011b-4fda-b725-58000a5b6b23","Type":"ContainerStarted","Data":"691ba8fb8930b41f7f59773dcf0d4779da9bb3d3ea1a7d48d2347d024994ea00"}
Oct 31 16:37:29.122237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:29.122200    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" event={"ID":"22cd68ee-011b-4fda-b725-58000a5b6b23","Type":"ContainerStarted","Data":"126ffe6be43be06e564df7418be6f71b2c7fbc4690328854e557d30e5dc5e192"}
Oct 31 16:37:29.122700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:29.122266    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:29.229038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:29.228987    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" podStartSLOduration=2.358142471 podStartE2EDuration="3.228973474s" totalImagesPullingTime="870.831003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:29.227510304 +0000 UTC m=+2094.289085492" watchObservedRunningTime="2026-10-31 16:37:29.228973474 +0000 UTC m=+2094.290548654"
Oct 31 16:37:32.946654 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:32.946599    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:37:35.136720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:35.136688    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:37:35.146910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:35.146880    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:37:35.248297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:35.248249    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:37:36.115953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.115922    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:37:36.117609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.116212    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy" containerID="cri-o://4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b" gracePeriod=30
Oct 31 16:37:36.119634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.119585    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:37:36.119830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.119806    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy" containerID="cri-o://391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc" gracePeriod=30
Oct 31 16:37:36.120873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.120842    4417 scope.go:122] "RemoveContainer" containerID="cc14e1c21871cab1051e9461458c4413325bbfda18a1e95bd0ce67c2ea5a3a36"
Oct 31 16:37:36.123571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.123550    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:37:36.123762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:36.123744    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy" containerID="cri-o://437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170" gracePeriod=30
Oct 31 16:37:40.147417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.147379    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" podUID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" containerName="payload-processing" containerID="cri-o://9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2" gracePeriod=25
Oct 31 16:37:40.248585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.248543    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" podUID="0470c18a-b968-40e7-bc2d-8092243c6719" containerName="payload-pre-processing" containerID="cri-o://8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a" gracePeriod=25
Oct 31 16:37:40.441862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.441832    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:37:40.483041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.483014    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\" (UniqueName: \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\") pod \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") "
Oct 31 16:37:40.483208 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.483102    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\") pod \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\" (UID: \"49e2b5ec-e928-455e-b1e8-af474a1a52f9\") "
Oct 31 16:37:40.483562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.483533    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume" pod "49e2b5ec-e928-455e-b1e8-af474a1a52f9" (UID: "49e2b5ec-e928-455e-b1e8-af474a1a52f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:40.483738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.483716    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e2b5ec-e928-455e-b1e8-af474a1a52f9-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:40.485291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.485272    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn" pod "49e2b5ec-e928-455e-b1e8-af474a1a52f9" (UID: "49e2b5ec-e928-455e-b1e8-af474a1a52f9"). InnerVolumeSpecName "kube-api-access-lr5mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:40.535321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.535295    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:37:40.584328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.584292    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\" (UniqueName: \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\") pod \"0470c18a-b968-40e7-bc2d-8092243c6719\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") "
Oct 31 16:37:40.584522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.584350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\") pod \"0470c18a-b968-40e7-bc2d-8092243c6719\" (UID: \"0470c18a-b968-40e7-bc2d-8092243c6719\") "
Oct 31 16:37:40.584775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.584748    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume" pod "0470c18a-b968-40e7-bc2d-8092243c6719" (UID: "0470c18a-b968-40e7-bc2d-8092243c6719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:40.585200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.585174    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0470c18a-b968-40e7-bc2d-8092243c6719-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:40.585304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.585204    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr5mn\" (UniqueName: \"kubernetes.io/projected/49e2b5ec-e928-455e-b1e8-af474a1a52f9-kube-api-access-lr5mn\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:40.587145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.587118    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp" pod "0470c18a-b968-40e7-bc2d-8092243c6719" (UID: "0470c18a-b968-40e7-bc2d-8092243c6719"). InnerVolumeSpecName "kube-api-access-pb2mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:40.687076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:40.686988    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2mp\" (UniqueName: \"kubernetes.io/projected/0470c18a-b968-40e7-bc2d-8092243c6719-kube-api-access-pb2mp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.222281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.217634    4417 generic.go:356] "Generic (PLEG): container finished" podID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" containerID="9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2" exitCode=0
Oct 31 16:37:41.222281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.217884    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" event={"ID":"49e2b5ec-e928-455e-b1e8-af474a1a52f9","Type":"ContainerDied","Data":"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"}
Oct 31 16:37:41.222281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.217950    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh" event={"ID":"49e2b5ec-e928-455e-b1e8-af474a1a52f9","Type":"ContainerDied","Data":"be56c874cc9340c93c02e09a1558089efe452494d067cb6a7cf185f341ce4ca9"}
Oct 31 16:37:41.222281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.217978    4417 scope.go:122] "RemoveContainer" containerID="9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"
Oct 31 16:37:41.222281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.218160    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"
Oct 31 16:37:41.269651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.262229    4417 generic.go:356] "Generic (PLEG): container finished" podID="0470c18a-b968-40e7-bc2d-8092243c6719" containerID="8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a" exitCode=0
Oct 31 16:37:41.269651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.262294    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" event={"ID":"0470c18a-b968-40e7-bc2d-8092243c6719","Type":"ContainerDied","Data":"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"}
Oct 31 16:37:41.269651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.262346    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth" event={"ID":"0470c18a-b968-40e7-bc2d-8092243c6719","Type":"ContainerDied","Data":"99561aa2065865e6a419e6823583e072bf3e37fa5b91560dbc945d8534a80689"}
Oct 31 16:37:41.269651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.262454    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"
Oct 31 16:37:41.324246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.324147    4417 scope.go:122] "RemoveContainer" containerID="9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"
Oct 31 16:37:41.325053 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:41.324869    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2\": container with ID starting with 9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2 not found: ID does not exist" containerID="9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"
Oct 31 16:37:41.325053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.324924    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2"} err="failed to get container status \"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2\": rpc error: code = NotFound desc = could not find container \"9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2\": container with ID starting with 9b4b3da1d60d6e5748f9312f24564b3f495df0221a04aff75aa5414cda3d98e2 not found: ID does not exist"
Oct 31 16:37:41.325053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.324950    4417 scope.go:122] "RemoveContainer" containerID="8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"
Oct 31 16:37:41.381730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.381695    4417 scope.go:122] "RemoveContainer" containerID="8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"
Oct 31 16:37:41.382815 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:41.382423    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a\": container with ID starting with 8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a not found: ID does not exist" containerID="8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"
Oct 31 16:37:41.382815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.382462    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a"} err="failed to get container status \"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a\": rpc error: code = NotFound desc = could not find container \"8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a\": container with ID starting with 8a420339a9059db7452dfd14b4946c583372148922dd303197f7d963b2652a8a not found: ID does not exist"
Oct 31 16:37:41.474354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.472739    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:37:41.476220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.476188    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-705a67-868484c-7r6wh"]
Oct 31 16:37:41.541453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.541423    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:37:41.548444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.548417    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-705a67-689d9d764c-jdfth"]
Oct 31 16:37:41.579956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.578922    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:37:41.605161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605058    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605161 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605125    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605161    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605220    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605309    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605345    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605467    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605540    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.605687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.605649    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\") pod \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\" (UID: \"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54\") "
Oct 31 16:37:41.606522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.606498    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.609287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.607869    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.609287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.608717    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.609287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.608869    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:41.609287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.609071    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.609287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.609259    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.609829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.609811    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.611574 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.611507    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "kube-api-access-fc892". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.611857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.611825    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:41.612043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.612026    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token" pod "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" (UID: "cbb1db46-e61a-4fa8-88ef-fffc69a9eb54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.618776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.618715    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:37:41.665487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.665460    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:37:41.710534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710351    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.710534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710404    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.710534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710521    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710558    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710641    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710661    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710686    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710727    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710799    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710807    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.710857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710832    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710878    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710913    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.710975    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711007    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711049    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711051    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711132    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711164    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711217    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711256    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\") pod \"134591d3-3561-4c21-96f7-f2f00b98f9ba\" (UID: \"134591d3-3561-4c21-96f7-f2f00b98f9ba\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711306    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\") pod \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\" (UID: \"1dea52a9-cfc5-434e-812f-bdacf803b5b2\") "
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711324    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711496    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711583    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.711799    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712044    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134591d3-3561-4c21-96f7-f2f00b98f9ba-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712065    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712079    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712093    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712106    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712120    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712134    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712148    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712162    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712175    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712189    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.712198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712205    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.713119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712220    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc892\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-kube-api-access-fc892\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.713119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712234    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.713119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712248    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.713119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712261    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.713119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.712579    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:37:41.713901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.713420    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.713901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.713752    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.715919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.715192    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:37:41.716823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.716758    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:41.716823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.716789    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "kube-api-access-llhrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.716823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.716796    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:37:41.717108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.717036    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "kube-api-access-kxmml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.717108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.717091    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token" pod "1dea52a9-cfc5-434e-812f-bdacf803b5b2" (UID: "1dea52a9-cfc5-434e-812f-bdacf803b5b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.718662 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.717646    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token" pod "134591d3-3561-4c21-96f7-f2f00b98f9ba" (UID: "134591d3-3561-4c21-96f7-f2f00b98f9ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:37:41.814046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814018    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814045    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814054    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814065    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814075    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhrj\" (UniqueName: \"kubernetes.io/projected/1dea52a9-cfc5-434e-812f-bdacf803b5b2-kube-api-access-llhrj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814085    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814095    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmml\" (UniqueName: \"kubernetes.io/projected/134591d3-3561-4c21-96f7-f2f00b98f9ba-kube-api-access-kxmml\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814104    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814113    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dea52a9-cfc5-434e-812f-bdacf803b5b2-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:41.814294 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:41.814122    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134591d3-3561-4c21-96f7-f2f00b98f9ba-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:37:42.271584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.271549    4417 generic.go:356] "Generic (PLEG): container finished" podID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerID="437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170" exitCode=0
Oct 31 16:37:42.272076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.271596    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"
Oct 31 16:37:42.272076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.271661    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" event={"ID":"1dea52a9-cfc5-434e-812f-bdacf803b5b2","Type":"ContainerDied","Data":"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"}
Oct 31 16:37:42.272076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.271715    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv" event={"ID":"1dea52a9-cfc5-434e-812f-bdacf803b5b2","Type":"ContainerDied","Data":"98603ae18814f888ad27ad947d224cc860e16b6920fae791e8638ea690b545fd"}
Oct 31 16:37:42.272076 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.271737    4417 scope.go:122] "RemoveContainer" containerID="437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"
Oct 31 16:37:42.273199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.273179    4417 generic.go:356] "Generic (PLEG): container finished" podID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerID="391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc" exitCode=0
Oct 31 16:37:42.273307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.273237    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"
Oct 31 16:37:42.273379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.273316    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" event={"ID":"134591d3-3561-4c21-96f7-f2f00b98f9ba","Type":"ContainerDied","Data":"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"}
Oct 31 16:37:42.273379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.273366    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c" event={"ID":"134591d3-3561-4c21-96f7-f2f00b98f9ba","Type":"ContainerDied","Data":"52a01fb39b7c731ba867fe66c041dd0c5e5d4a32909d7ee63ae91e582ea1a716"}
Oct 31 16:37:42.275000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.274979    4417 generic.go:356] "Generic (PLEG): container finished" podID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerID="4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b" exitCode=0
Oct 31 16:37:42.275188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.275014    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" event={"ID":"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54","Type":"ContainerDied","Data":"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"}
Oct 31 16:37:42.275326 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.275309    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn" event={"ID":"cbb1db46-e61a-4fa8-88ef-fffc69a9eb54","Type":"ContainerDied","Data":"15d7ca1fa94978a452acfb145fe6f6d7f0da3bc71b1df9882ab3b7e14a99e574"}
Oct 31 16:37:42.275423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.275044    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"
Oct 31 16:37:42.283334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.283315    4417 scope.go:122] "RemoveContainer" containerID="437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"
Oct 31 16:37:42.283602 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:42.283582    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170\": container with ID starting with 437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170 not found: ID does not exist" containerID="437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"
Oct 31 16:37:42.283706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.283607    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170"} err="failed to get container status \"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170\": rpc error: code = NotFound desc = could not find container \"437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170\": container with ID starting with 437ca0e932426d76820c676bfcea5caf931331e45853236dc4d980ca00979170 not found: ID does not exist"
Oct 31 16:37:42.283706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.283651    4417 scope.go:122] "RemoveContainer" containerID="391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"
Oct 31 16:37:42.293203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.293187    4417 scope.go:122] "RemoveContainer" containerID="391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"
Oct 31 16:37:42.293447 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:42.293432    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc\": container with ID starting with 391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc not found: ID does not exist" containerID="391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"
Oct 31 16:37:42.293500 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.293452    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc"} err="failed to get container status \"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc\": rpc error: code = NotFound desc = could not find container \"391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc\": container with ID starting with 391e942722422e414084381bc0c70d312cb13cc882a48d3c7498f55bee2a37cc not found: ID does not exist"
Oct 31 16:37:42.293500 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.293468    4417 scope.go:122] "RemoveContainer" containerID="4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"
Oct 31 16:37:42.303491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.303472    4417 scope.go:122] "RemoveContainer" containerID="4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"
Oct 31 16:37:42.303777 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:37:42.303757    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b\": container with ID starting with 4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b not found: ID does not exist" containerID="4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"
Oct 31 16:37:42.303853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.303783    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b"} err="failed to get container status \"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b\": rpc error: code = NotFound desc = could not find container \"4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b\": container with ID starting with 4934d1e7a6683181e703d1a299def96144e913c1991ec8ebb251cf9cdd3f424b not found: ID does not exist"
Oct 31 16:37:42.424175 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.424136    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:37:42.469864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.469773    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-t8f9c"]
Oct 31 16:37:42.527951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.527897    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:37:42.531817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.531781    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-l7bmv"]
Oct 31 16:37:42.632084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.632048    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:37:42.658726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:42.657029    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-gp2jn"]
Oct 31 16:37:43.086847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:43.086809    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0470c18a-b968-40e7-bc2d-8092243c6719" path="/var/lib/kubelet/pods/0470c18a-b968-40e7-bc2d-8092243c6719/volumes"
Oct 31 16:37:43.087508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:43.087485    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" path="/var/lib/kubelet/pods/134591d3-3561-4c21-96f7-f2f00b98f9ba/volumes"
Oct 31 16:37:43.088102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:43.088083    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" path="/var/lib/kubelet/pods/1dea52a9-cfc5-434e-812f-bdacf803b5b2/volumes"
Oct 31 16:37:43.091261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:43.091235    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" path="/var/lib/kubelet/pods/49e2b5ec-e928-455e-b1e8-af474a1a52f9/volumes"
Oct 31 16:37:43.091896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:43.091875    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" path="/var/lib/kubelet/pods/cbb1db46-e61a-4fa8-88ef-fffc69a9eb54/volumes"
Oct 31 16:37:52.424269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.424189    4417 generic.go:356] "Generic (PLEG): container finished" podID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerID="a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be" exitCode=0
Oct 31 16:37:52.424686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.424270    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerDied","Data":"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"}
Oct 31 16:37:52.424686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.424672    4417 scope.go:122] "RemoveContainer" containerID="a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"
Oct 31 16:37:52.426428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.426401    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:37:52.426512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.426472    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:37:52.426512 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:52.426501    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:37:53.432888 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:53.432856    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerStarted","Data":"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"}
Oct 31 16:37:54.051640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:54.051156    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:37:54.352775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:54.352675    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:37:54.353756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:54.353705    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:37:54.353970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:54.353947    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:37:55.356641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:55.348695    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:37:55.356641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:55.348757    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:37:56.350641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:56.349889    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:37:56.350641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:56.349944    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:37:57.354639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:57.353957    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:37:57.354639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:57.354031    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:37:58.364789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:58.364651    4417 patch_prober.go:29] interesting pod/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 16:37:58.364789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:58.364720    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 16:37:59.354170 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:59.354141    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:37:59.355319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:59.355295    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:37:59.698782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:59.698699    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:37:59.699180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:37:59.698936    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" podUID="8d2dfe63-3572-49c6-86af-6316c24470b7" containerName="maas-api" containerID="cri-o://14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3" gracePeriod=30
Oct 31 16:38:00.027854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.027826    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:38:00.192041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192010    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\") pod \"8d2dfe63-3572-49c6-86af-6316c24470b7\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") "
Oct 31 16:38:00.192041 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192052    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\") pod \"8d2dfe63-3572-49c6-86af-6316c24470b7\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") "
Oct 31 16:38:00.192328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192102    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\" (UniqueName: \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\") pod \"8d2dfe63-3572-49c6-86af-6316c24470b7\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") "
Oct 31 16:38:00.192457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192391    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") pod \"8d2dfe63-3572-49c6-86af-6316c24470b7\" (UID: \"8d2dfe63-3572-49c6-86af-6316c24470b7\") "
Oct 31 16:38:00.192457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192443    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca" pod "8d2dfe63-3572-49c6-86af-6316c24470b7" (UID: "8d2dfe63-3572-49c6-86af-6316c24470b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:00.192912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.192890    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dfe63-3572-49c6-86af-6316c24470b7-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:00.195127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.195098    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls" pod "8d2dfe63-3572-49c6-86af-6316c24470b7" (UID: "8d2dfe63-3572-49c6-86af-6316c24470b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:00.195127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.195115    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h" pod "8d2dfe63-3572-49c6-86af-6316c24470b7" (UID: "8d2dfe63-3572-49c6-86af-6316c24470b7"). InnerVolumeSpecName "kube-api-access-wg92h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:00.195472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.195456    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs" pod "8d2dfe63-3572-49c6-86af-6316c24470b7" (UID: "8d2dfe63-3572-49c6-86af-6316c24470b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:00.303641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.300370    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:00.303641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.300408    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2dfe63-3572-49c6-86af-6316c24470b7-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:00.303641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.300425    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg92h\" (UniqueName: \"kubernetes.io/projected/8d2dfe63-3572-49c6-86af-6316c24470b7-kube-api-access-wg92h\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:00.509445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.509403    4417 generic.go:356] "Generic (PLEG): container finished" podID="8d2dfe63-3572-49c6-86af-6316c24470b7" containerID="14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3" exitCode=0
Oct 31 16:38:00.509625 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.509449    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"
Oct 31 16:38:00.509625 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.509491    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" event={"ID":"8d2dfe63-3572-49c6-86af-6316c24470b7","Type":"ContainerDied","Data":"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"}
Oct 31 16:38:00.509625 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.509547    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j" event={"ID":"8d2dfe63-3572-49c6-86af-6316c24470b7","Type":"ContainerDied","Data":"50a2c5bc64eec3fb72c62a49546032aae1b824ae81ae5ec946ba91e3ba63c421"}
Oct 31 16:38:00.509625 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.509576    4417 scope.go:122] "RemoveContainer" containerID="14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"
Oct 31 16:38:00.520491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.520473    4417 scope.go:122] "RemoveContainer" containerID="14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"
Oct 31 16:38:00.520814 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:38:00.520794    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3\": container with ID starting with 14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3 not found: ID does not exist" containerID="14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"
Oct 31 16:38:00.520900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.520820    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3"} err="failed to get container status \"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3\": rpc error: code = NotFound desc = could not find container \"14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3\": container with ID starting with 14a00f2657ddc7e2b211485a122867f555f70bf4b4f2c5f54e28d4528f4dc8b3 not found: ID does not exist"
Oct 31 16:38:00.671688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.671558    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:38:00.740212 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:00.740177    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dac4fa-699589cb7b-5x84j"]
Oct 31 16:38:01.051172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:01.051142    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2dfe63-3572-49c6-86af-6316c24470b7" path="/var/lib/kubelet/pods/8d2dfe63-3572-49c6-86af-6316c24470b7/volumes"
Oct 31 16:38:03.004221 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:38:03.003280    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:38:06.357452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.357420    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:38:06.358222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358200    4417 cpu_manager.go:414] "containerMap: removing container" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy"
Oct 31 16:38:06.358222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358222    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358244    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0470c18a-b968-40e7-bc2d-8092243c6719" containerName="payload-pre-processing"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358249    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0470c18a-b968-40e7-bc2d-8092243c6719" containerName="payload-pre-processing"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358275    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8d2dfe63-3572-49c6-86af-6316c24470b7" containerName="maas-api"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358280    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2dfe63-3572-49c6-86af-6316c24470b7" containerName="maas-api"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358293    4417 cpu_manager.go:414] "containerMap: removing container" podUID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" containerName="payload-processing"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358297    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" containerName="payload-processing"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358312    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358317    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358329    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy"
Oct 31 16:38:06.358380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358334    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358417    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="134591d3-3561-4c21-96f7-f2f00b98f9ba" containerName="istio-proxy"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358427    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0470c18a-b968-40e7-bc2d-8092243c6719" containerName="payload-pre-processing"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358438    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2dfe63-3572-49c6-86af-6316c24470b7" containerName="maas-api"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358445    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e2b5ec-e928-455e-b1e8-af474a1a52f9" containerName="payload-processing"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358452    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dea52a9-cfc5-434e-812f-bdacf803b5b2" containerName="istio-proxy"
Oct 31 16:38:06.358719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.358458    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb1db46-e61a-4fa8-88ef-fffc69a9eb54" containerName="istio-proxy"
Oct 31 16:38:06.361906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.361887    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418177    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418221    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418321    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418349    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5n8v\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418435    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418463    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418500    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418654    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.418722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.418694    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519511    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519556    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519592    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519661    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519694    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519784    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.519823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519811    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519887    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.519931    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5n8v\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.520003    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.520053    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.520174    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.520392    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.520679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.520656    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.522263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.522240    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.522449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.522427    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.554584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.554492    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:38:06.572359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.572324    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5n8v\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.573160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.573123    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:06.675380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:06.675293    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:07.094245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.094216    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:38:07.096661 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:07.096631    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f30e01f_316d_42db_a49f_e76808bf20fe.slice/crio-90876a7fefab6bdfde836640f3aff88e85770e9aebc58830da8b549e9dd52d31 WatchSource:0}: Error finding container 90876a7fefab6bdfde836640f3aff88e85770e9aebc58830da8b549e9dd52d31: Status 404 returned error can't find the container with id 90876a7fefab6bdfde836640f3aff88e85770e9aebc58830da8b549e9dd52d31
Oct 31 16:38:07.099273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.099238    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.099358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.099304    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.099358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.099333    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:07.573562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.573521    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" event={"ID":"6f30e01f-316d-42db-a49f-e76808bf20fe","Type":"ContainerStarted","Data":"144d71ab8854ce3c619494dbd7fb733aea992618f812cbf8190dc0726c17b7bc"}
Oct 31 16:38:07.574100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.573592    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" event={"ID":"6f30e01f-316d-42db-a49f-e76808bf20fe","Type":"ContainerStarted","Data":"90876a7fefab6bdfde836640f3aff88e85770e9aebc58830da8b549e9dd52d31"}
Oct 31 16:38:07.675820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.675779    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:07.683763 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:07.683713    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podStartSLOduration=1.683698686 podStartE2EDuration="1.683698686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:07.680824587 +0000 UTC m=+2132.742399774" watchObservedRunningTime="2026-10-31 16:38:07.683698686 +0000 UTC m=+2132.745273928"
Oct 31 16:38:08.677305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:08.677265    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 16:38:08.677800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:08.677329    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 16:38:09.676311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:09.676271    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 16:38:09.676528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:09.676347    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 16:38:10.683638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:10.683184    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 16:38:10.683638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:10.683273    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 16:38:11.676441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:11.676403    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 16:38:11.676682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:11.676478    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 16:38:12.676078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:12.676036    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 16:38:12.676488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:12.676110    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 16:38:13.717737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:13.717701    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:13.718190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:13.718010    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:13.719143 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:13.719115    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:38:21.484468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.484427    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:38:21.489110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.489088    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616296    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616348    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616371    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616586    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616667    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616792    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616918    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvsp\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.616995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.616949    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.617047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.617026    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.689216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.689147    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:38:21.718454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718419    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvsp\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.718694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718460    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.718694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718514    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.718694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718557    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.718694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718576    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.718694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718594    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718703    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718740    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.718778    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.719002    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.719506    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.719582    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.719691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.719676    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.720031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.720005    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.721393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.721368    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.721533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.721515    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.770714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.770641    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.770925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.770902    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvsp\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.803675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.803646    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:21.862818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:21.862779    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:38:22.240395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.240363    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:38:22.241569 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:22.241542    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9917d9f7_bdc6_4be5_a008_d0d8ad96254d.slice/crio-b7dfc75fd4e9173568014aa0e7510a7c520a4078260fa0350f3a35ca8fe6f480 WatchSource:0}: Error finding container b7dfc75fd4e9173568014aa0e7510a7c520a4078260fa0350f3a35ca8fe6f480: Status 404 returned error can't find the container with id b7dfc75fd4e9173568014aa0e7510a7c520a4078260fa0350f3a35ca8fe6f480
Oct 31 16:38:22.244735 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.244702    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.244819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.244769    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.244819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.244798    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:22.865056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.865011    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" event={"ID":"9917d9f7-bdc6-4be5-a008-d0d8ad96254d","Type":"ContainerStarted","Data":"e2d9c19bf53fe878308183686c059b23d7246c983fb6b88912aa17303f8a0bf1"}
Oct 31 16:38:22.865536 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.865084    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" event={"ID":"9917d9f7-bdc6-4be5-a008-d0d8ad96254d","Type":"ContainerStarted","Data":"b7dfc75fd4e9173568014aa0e7510a7c520a4078260fa0350f3a35ca8fe6f480"}
Oct 31 16:38:22.979858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:22.979798    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podStartSLOduration=1.979779382 podStartE2EDuration="1.979779382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:22.974386408 +0000 UTC m=+2148.035961596" watchObservedRunningTime="2026-10-31 16:38:22.979779382 +0000 UTC m=+2148.041354569"
Oct 31 16:38:23.804814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:23.804764    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:23.806610 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:23.806584    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:23.806781 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:23.806671    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:24.805034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:24.804997    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:24.805444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:24.805053    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:25.804515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:25.804480    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:25.804722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:25.804548    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:26.804741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:26.804701    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:26.805202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:26.804776    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:26.863747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:26.863685    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" podUID="c9d61af7-0e80-46f0-b412-742398967894" containerName="payload-processing" containerID="cri-o://b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b" gracePeriod=25
Oct 31 16:38:27.209327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.209296    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:38:27.267297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.267255    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\") pod \"c9d61af7-0e80-46f0-b412-742398967894\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") "
Oct 31 16:38:27.267490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.267464    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\" (UniqueName: \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\") pod \"c9d61af7-0e80-46f0-b412-742398967894\" (UID: \"c9d61af7-0e80-46f0-b412-742398967894\") "
Oct 31 16:38:27.267719 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.267689    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume" pod "c9d61af7-0e80-46f0-b412-742398967894" (UID: "c9d61af7-0e80-46f0-b412-742398967894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:27.268123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.268100    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d61af7-0e80-46f0-b412-742398967894-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:27.269758 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.269736    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd" pod "c9d61af7-0e80-46f0-b412-742398967894" (UID: "c9d61af7-0e80-46f0-b412-742398967894"). InnerVolumeSpecName "kube-api-access-scgnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:27.377693 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.373339    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgnd\" (UniqueName: \"kubernetes.io/projected/c9d61af7-0e80-46f0-b412-742398967894-kube-api-access-scgnd\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:27.804307 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.804275    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:27.804488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.804336    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:27.903468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.903430    4417 generic.go:356] "Generic (PLEG): container finished" podID="c9d61af7-0e80-46f0-b412-742398967894" containerID="b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b" exitCode=0
Oct 31 16:38:27.903892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.903477    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"
Oct 31 16:38:27.903892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.903510    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" event={"ID":"c9d61af7-0e80-46f0-b412-742398967894","Type":"ContainerDied","Data":"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"}
Oct 31 16:38:27.903892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.903566    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9" event={"ID":"c9d61af7-0e80-46f0-b412-742398967894","Type":"ContainerDied","Data":"c7ab64e2a22c1c8cddd3c887c022515e29ddda5a31dffd41a05dc284b4f76444"}
Oct 31 16:38:27.903892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.903591    4417 scope.go:122] "RemoveContainer" containerID="b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"
Oct 31 16:38:27.914771 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.914752    4417 scope.go:122] "RemoveContainer" containerID="b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"
Oct 31 16:38:27.915009 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:38:27.914988    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b\": container with ID starting with b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b not found: ID does not exist" containerID="b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"
Oct 31 16:38:27.915079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:27.915021    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b"} err="failed to get container status \"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b\": rpc error: code = NotFound desc = could not find container \"b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b\": container with ID starting with b6c2fd45ef4343da0b7ea72c9cee3f0cdf55f050cbf38dc860c7739e03aeea7b not found: ID does not exist"
Oct 31 16:38:28.072971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:28.070167    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:38:28.074738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:28.074707    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3dc1da-6b9b59d4c7-wfxb9"]
Oct 31 16:38:28.819454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:28.819413    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Oct 31 16:38:28.819766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:28.819735    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Oct 31 16:38:29.053173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:29.053140    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d61af7-0e80-46f0-b412-742398967894" path="/var/lib/kubelet/pods/c9d61af7-0e80-46f0-b412-742398967894/volumes"
Oct 31 16:38:29.808953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:29.808911    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:29.809411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:29.809390    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:29.810171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:29.810146    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:38:32.975924 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:38:32.975883    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:38:34.871927 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.871880    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:38:34.873067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.873040    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c9d61af7-0e80-46f0-b412-742398967894" containerName="payload-processing"
Oct 31 16:38:34.873167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.873069    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d61af7-0e80-46f0-b412-742398967894" containerName="payload-processing"
Oct 31 16:38:34.873250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.873235    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d61af7-0e80-46f0-b412-742398967894" containerName="payload-processing"
Oct 31 16:38:34.878743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.878715    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:34.941676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:34.939814    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-66db1c-openshift-default-dockercfg-h78j7\""
Oct 31 16:38:35.128694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.128581    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.129075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.129049    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46ql\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.129231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.129217    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.129347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.129334    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.129454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.129442    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.134204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.129589    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.134457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.134433    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.134701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.134682    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.134840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.134823    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.155316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.155284    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303292    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303343    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46ql\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303373    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303404    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303437    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303495    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303537    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303642    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.303669    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.305199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.304037    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.306791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.306438    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.309465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.308972    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.309465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.309198    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.309465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.309401    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.318206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.318179    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.322960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.322755    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.372937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.372857    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.377709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.377675    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46ql\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\") pod \"e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") " pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:35.591189 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.591156    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-66db1c-openshift-default-dockercfg-h78j7\""
Oct 31 16:38:35.594265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:35.594243    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:36.075018 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:36.074980    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb438b8f_863e_4720_b14e_6d4bae5f151b.slice/crio-14bbe7fcdc588188626fa5bb12b4ba0bcc2980c94a53fba9f827a6a5fa2c2dce WatchSource:0}: Error finding container 14bbe7fcdc588188626fa5bb12b4ba0bcc2980c94a53fba9f827a6a5fa2c2dce: Status 404 returned error can't find the container with id 14bbe7fcdc588188626fa5bb12b4ba0bcc2980c94a53fba9f827a6a5fa2c2dce
Oct 31 16:38:36.079760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.079733    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:38:36.080607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.080569    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:36.080683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.080668    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:36.080728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.080713    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:36.224200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.223917    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerStarted","Data":"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"}
Oct 31 16:38:36.224200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.223996    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerStarted","Data":"14bbe7fcdc588188626fa5bb12b4ba0bcc2980c94a53fba9f827a6a5fa2c2dce"}
Oct 31 16:38:36.334362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.334248    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podStartSLOduration=2.334229374 podStartE2EDuration="2.334229374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:36.333935958 +0000 UTC m=+2161.395511157" watchObservedRunningTime="2026-10-31 16:38:36.334229374 +0000 UTC m=+2161.395804560"
Oct 31 16:38:36.599920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.599823    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:36.605130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.605097    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:38:36.611418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.611392    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.685411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.685373    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:38:36.724915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.724869    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.724947    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725010    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725046    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725174    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86l6k\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725220    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725328 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725322    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725354    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.725497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.725459    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.826999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.826955    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.826999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827006    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827083    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827133    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827188    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827247    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827277    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827315    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86l6k\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827358    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827524    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827560    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827639    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.827806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.827794    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.828229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.828205    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.829855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.829827    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.830707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.830679    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.880642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.880548    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.880748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.880701    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86l6k\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:36.930148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:36.930099    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:37.370783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:37.370745    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:38:37.373935 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:37.373899    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab92634_4b15_409c_abb0_c45ff14e38f8.slice/crio-303b69a0dbf26466fb4878b4314f4e06549b49b494e9e4d2a691f8819462627c WatchSource:0}: Error finding container 303b69a0dbf26466fb4878b4314f4e06549b49b494e9e4d2a691f8819462627c: Status 404 returned error can't find the container with id 303b69a0dbf26466fb4878b4314f4e06549b49b494e9e4d2a691f8819462627c
Oct 31 16:38:37.377221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:37.377177    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:37.377336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:37.377253    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:37.377336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:37.377283    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:37.600975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:37.600944    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:38.240003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.239941    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" event={"ID":"6ab92634-4b15-409c-abb0-c45ff14e38f8","Type":"ContainerStarted","Data":"577486bbd4113722bee1a54c4bf74c5b8ba865951f432df52e3750b936a1bd7c"}
Oct 31 16:38:38.240230 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.240024    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" event={"ID":"6ab92634-4b15-409c-abb0-c45ff14e38f8","Type":"ContainerStarted","Data":"303b69a0dbf26466fb4878b4314f4e06549b49b494e9e4d2a691f8819462627c"}
Oct 31 16:38:38.240298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.240262    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:38.241565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.241539    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:38:38.352458 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.352380    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podStartSLOduration=2.352356998 podStartE2EDuration="2.352356998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:38.348214013 +0000 UTC m=+2163.409789200" watchObservedRunningTime="2026-10-31 16:38:38.352356998 +0000 UTC m=+2163.413932185"
Oct 31 16:38:38.931252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.931211    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:38.932786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.932756    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:38.932917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:38.932810    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:39.931167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:39.931118    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:39.931362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:39.931192    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:40.931264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:40.931229    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:40.931439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:40.931288    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:41.931186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:41.931143    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:41.931366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:41.931200    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:42.188173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.185739    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:43546->10.133.0.152:15021: read: connection reset by peer" start-of-body=
Oct 31 16:38:42.188173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.185800    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": read tcp 10.133.0.2:43546->10.133.0.152:15021: read: connection reset by peer"
Oct 31 16:38:42.315303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.315268    4417 generic.go:356] "Generic (PLEG): container finished" podID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerID="33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e" exitCode=0
Oct 31 16:38:42.315519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.315364    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerDied","Data":"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"}
Oct 31 16:38:42.315854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.315830    4417 scope.go:122] "RemoveContainer" containerID="33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"
Oct 31 16:38:42.317895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.317851    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:42.317997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.317930    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:42.317997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.317973    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:38:42.930899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.930857    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:42.931093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:42.930921    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:43.327004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.326960    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerStarted","Data":"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"}
Oct 31 16:38:43.476202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.476165    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:38:43.476947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.476917    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:43.477176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.477150    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:43.930692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.930657    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 16:38:43.930920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:43.930719    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 16:38:44.476649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:44.476121    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:44.476649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:44.476186    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:45.703640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.703285    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:45.703640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.703345    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:45.715819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.715789    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:38:45.732717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.732553    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.732717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.732602    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:45.733034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.732999    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:38:45.749155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.734920    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:45.749155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.738423    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:38:45.749155 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.748701    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:38:45.776811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.776323    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:38:45.776811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.776450    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:45.827734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.827706    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-66db1c-dockercfg-hmfrh\""
Oct 31 16:38:45.827933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.827797    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-66db1c\""
Oct 31 16:38:45.828107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.828089    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.828252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.828235    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:45.828516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.828491    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjth\" (UniqueName: \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:45.828582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.828566    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vs5\" (UniqueName: \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.930125 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.930083    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjth\" (UniqueName: \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:45.930301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.930162    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vs5\" (UniqueName: \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.930360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.930302    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.930417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.930385    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:45.931137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.931108    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:45.931277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:45.931256    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:46.024701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.009739    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vs5\" (UniqueName: \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\") pod \"payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:46.043047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.043013    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjth\" (UniqueName: \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\") pod \"payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") " pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:46.093572 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.093536    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:46.100922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.100900    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:46.476630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.476539    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:46.476940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.476899    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:46.511855 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.511808    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:38:46.512590 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:46.512553    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308e9eb2_2978_440c_920f_a6950276037d.slice/crio-9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672 WatchSource:0}: Error finding container 9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672: Status 404 returned error can't find the container with id 9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672
Oct 31 16:38:46.573885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.573827    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:38:46.576920 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:38:46.575761    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eabd75_ed00_4313_8f44_ebe7f6faf756.slice/crio-30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e WatchSource:0}: Error finding container 30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e: Status 404 returned error can't find the container with id 30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e
Oct 31 16:38:46.732772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.732674    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" event={"ID":"26eabd75-ed00-4313-8f44-ebe7f6faf756","Type":"ContainerStarted","Data":"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"}
Oct 31 16:38:46.732772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.732736    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" event={"ID":"26eabd75-ed00-4313-8f44-ebe7f6faf756","Type":"ContainerStarted","Data":"30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e"}
Oct 31 16:38:46.732772 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.732770    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:46.769643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.753367    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" event={"ID":"308e9eb2-2978-440c-920f-a6950276037d","Type":"ContainerStarted","Data":"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"}
Oct 31 16:38:46.769991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.769951    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" event={"ID":"308e9eb2-2978-440c-920f-a6950276037d","Type":"ContainerStarted","Data":"9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672"}
Oct 31 16:38:46.770138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.770120    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:46.878459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:46.878399    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" podStartSLOduration=1.878379159 podStartE2EDuration="1.878379159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:46.861944238 +0000 UTC m=+2171.923519425" watchObservedRunningTime="2026-10-31 16:38:46.878379159 +0000 UTC m=+2171.939954345"
Oct 31 16:38:47.012416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:47.010976    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" podStartSLOduration=2.010955309 podStartE2EDuration="2.010955309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:46.980118764 +0000 UTC m=+2172.041693976" watchObservedRunningTime="2026-10-31 16:38:47.010955309 +0000 UTC m=+2172.072530496"
Oct 31 16:38:47.476383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:47.476322    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:47.476583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:47.476407    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:48.475829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:48.475784    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 16:38:48.476325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:48.475847    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 16:38:49.522079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:49.522041    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:38:49.522555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:49.522520    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:38:49.523369 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:49.523347    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:38:50.437650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:50.436153    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:38:50.437650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:50.436506    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" podUID="61214a6c-c042-4c39-b547-429158e529df" containerName="maas-api" containerID="cri-o://ec437d91b985c5faaa88d881c3522bdf72d8027336c4a13b094e430aa08c442b" gracePeriod=30
Oct 31 16:38:50.794234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:50.794196    4417 generic.go:356] "Generic (PLEG): container finished" podID="61214a6c-c042-4c39-b547-429158e529df" containerID="ec437d91b985c5faaa88d881c3522bdf72d8027336c4a13b094e430aa08c442b" exitCode=0
Oct 31 16:38:50.794751 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:50.794290    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" event={"ID":"61214a6c-c042-4c39-b547-429158e529df","Type":"ContainerDied","Data":"ec437d91b985c5faaa88d881c3522bdf72d8027336c4a13b094e430aa08c442b"}
Oct 31 16:38:50.983520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:50.983488    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:38:51.055663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.053041    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\" (UniqueName: \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\") pod \"61214a6c-c042-4c39-b547-429158e529df\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") "
Oct 31 16:38:51.055663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.053128    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\") pod \"61214a6c-c042-4c39-b547-429158e529df\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") "
Oct 31 16:38:51.055663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.053185    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") pod \"61214a6c-c042-4c39-b547-429158e529df\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") "
Oct 31 16:38:51.055663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.053350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\") pod \"61214a6c-c042-4c39-b547-429158e529df\" (UID: \"61214a6c-c042-4c39-b547-429158e529df\") "
Oct 31 16:38:51.055663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.055102    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca" pod "61214a6c-c042-4c39-b547-429158e529df" (UID: "61214a6c-c042-4c39-b547-429158e529df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:38:51.081006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.080832    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw" pod "61214a6c-c042-4c39-b547-429158e529df" (UID: "61214a6c-c042-4c39-b547-429158e529df"). InnerVolumeSpecName "kube-api-access-928cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:38:51.101609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.101195    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls" pod "61214a6c-c042-4c39-b547-429158e529df" (UID: "61214a6c-c042-4c39-b547-429158e529df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:51.101609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.104192    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs" pod "61214a6c-c042-4c39-b547-429158e529df" (UID: "61214a6c-c042-4c39-b547-429158e529df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:38:51.158691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.157062    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-928cw\" (UniqueName: \"kubernetes.io/projected/61214a6c-c042-4c39-b547-429158e529df-kube-api-access-928cw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:51.158691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.157100    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:51.158691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.157113    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61214a6c-c042-4c39-b547-429158e529df-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:51.158691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.157129    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61214a6c-c042-4c39-b547-429158e529df-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:38:51.802755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.802719    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"
Oct 31 16:38:51.802755 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.802748    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb" event={"ID":"61214a6c-c042-4c39-b547-429158e529df","Type":"ContainerDied","Data":"5fceaed4c228e0ec732e10ac286dd26cce3f22c3c3877def02c93ea60524a11a"}
Oct 31 16:38:51.803268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.802813    4417 scope.go:122] "RemoveContainer" containerID="ec437d91b985c5faaa88d881c3522bdf72d8027336c4a13b094e430aa08c442b"
Oct 31 16:38:51.955011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:51.954964    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:38:52.000961 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:52.000924    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-705a67-dc8dbf7c-xjtdb"]
Oct 31 16:38:53.051530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:53.051494    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61214a6c-c042-4c39-b547-429158e529df" path="/var/lib/kubelet/pods/61214a6c-c042-4c39-b547-429158e529df/volumes"
Oct 31 16:38:57.791340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:57.791303    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:38:57.793549 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:57.793437    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:38:59.424085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:59.424039    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:38:59.424665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:38:59.424323    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy" containerID="cri-o://1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5" gracePeriod=30
Oct 31 16:39:02.949251 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:02.949217    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:39:04.489077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.488562    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:39:04.589069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.589027    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:39:04.731501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.731471    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:39:04.794816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.794774    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.794915    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.794986    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795017    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795054    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795087    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795298 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795224    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795326    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.795434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.795415    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\") pod \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\" (UID: \"fdc3cb4c-9227-4c52-a024-b9564d7d06e4\") "
Oct 31 16:39:04.797295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.796652    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:04.797295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.796840    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:04.797295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.797015    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:04.797295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.797166    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:04.797694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.797443    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:04.802410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.802380    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "kube-api-access-c9z5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:04.806314 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.806275    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:04.806571 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.806533    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:04.807303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.807280    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo" pod "fdc3cb4c-9227-4c52-a024-b9564d7d06e4" (UID: "fdc3cb4c-9227-4c52-a024-b9564d7d06e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:04.898338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898294    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898338    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9z5w\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-kube-api-access-c9z5w\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898355    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898368    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898382    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898395    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898408    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898421    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.898676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.898435    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdc3cb4c-9227-4c52-a024-b9564d7d06e4-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:04.901566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.901531    4417 generic.go:356] "Generic (PLEG): container finished" podID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerID="1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5" exitCode=0
Oct 31 16:39:04.901718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.901600    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerDied","Data":"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"}
Oct 31 16:39:04.901718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.901668    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr" event={"ID":"fdc3cb4c-9227-4c52-a024-b9564d7d06e4","Type":"ContainerDied","Data":"5a4de262da210b6e4f6affdad16a9fabae1a2a5701e65910fc6909b5ace2029b"}
Oct 31 16:39:04.901718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.901701    4417 scope.go:122] "RemoveContainer" containerID="1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"
Oct 31 16:39:04.901911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.901890    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"
Oct 31 16:39:04.930651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.930591    4417 scope.go:122] "RemoveContainer" containerID="ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"
Oct 31 16:39:04.942686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.942665    4417 scope.go:122] "RemoveContainer" containerID="1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"
Oct 31 16:39:04.942978 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:04.942951    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5\": container with ID starting with 1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5 not found: ID does not exist" containerID="1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"
Oct 31 16:39:04.943067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.942995    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5"} err="failed to get container status \"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5\": rpc error: code = NotFound desc = could not find container \"1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5\": container with ID starting with 1254d5846310686e68ddc7b9349afd7b4b7eb67ff36ace2b31bf767ee48522d5 not found: ID does not exist"
Oct 31 16:39:04.943067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.943021    4417 scope.go:122] "RemoveContainer" containerID="ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"
Oct 31 16:39:04.943344 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:04.943325    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea\": container with ID starting with ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea not found: ID does not exist" containerID="ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"
Oct 31 16:39:04.943414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:04.943351    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea"} err="failed to get container status \"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea\": rpc error: code = NotFound desc = could not find container \"ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea\": container with ID starting with ee43a1b01cbe40dbf806f87c1b5da844e99f7742dbceb72adc1b6a10231d55ea not found: ID does not exist"
Oct 31 16:39:05.120167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:05.120133    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:39:05.123360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:05.123333    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-705a67-openshift-default-85c774bd65-pnlgr"]
Oct 31 16:39:06.812485 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:06.812443    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:39:06.909919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:06.909876    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:39:07.052332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:07.052295    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" path="/var/lib/kubelet/pods/fdc3cb4c-9227-4c52-a024-b9564d7d06e4/volumes"
Oct 31 16:39:09.489331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.489277    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" podUID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" containerName="payload-processing" containerID="cri-o://7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412" gracePeriod=25
Oct 31 16:39:09.590503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.590455    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" podUID="d3399c25-2c2d-43e8-aba3-36b6350615ab" containerName="payload-pre-processing" containerID="cri-o://6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd" gracePeriod=25
Oct 31 16:39:09.799768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.799734    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:39:09.895243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.895215    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:39:09.901818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.901787    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\") pod \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") "
Oct 31 16:39:09.901996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.901944    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\" (UniqueName: \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\") pod \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\" (UID: \"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed\") "
Oct 31 16:39:09.902152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.902130    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume" pod "32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" (UID: "32b7bc81-0b35-46f1-aea3-1d8d417ee2ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:09.902479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.902451    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:09.903969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.903947    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz" pod "32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" (UID: "32b7bc81-0b35-46f1-aea3-1d8d417ee2ed"). InnerVolumeSpecName "kube-api-access-jmsfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:09.993045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.993010    4417 generic.go:356] "Generic (PLEG): container finished" podID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" containerID="7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412" exitCode=0
Oct 31 16:39:09.993250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.993090    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"
Oct 31 16:39:09.993250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.993089    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" event={"ID":"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed","Type":"ContainerDied","Data":"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"}
Oct 31 16:39:09.993250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.993238    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx" event={"ID":"32b7bc81-0b35-46f1-aea3-1d8d417ee2ed","Type":"ContainerDied","Data":"c9b5d77905dfb57540e66720e0e32cf81ff35a6fdf189b44d380e42d86831133"}
Oct 31 16:39:09.993415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.993262    4417 scope.go:122] "RemoveContainer" containerID="7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"
Oct 31 16:39:09.994963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.994893    4417 generic.go:356] "Generic (PLEG): container finished" podID="d3399c25-2c2d-43e8-aba3-36b6350615ab" containerID="6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd" exitCode=0
Oct 31 16:39:09.994963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.994915    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" event={"ID":"d3399c25-2c2d-43e8-aba3-36b6350615ab","Type":"ContainerDied","Data":"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"}
Oct 31 16:39:09.994963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.994959    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"
Oct 31 16:39:09.995181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:09.994983    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2" event={"ID":"d3399c25-2c2d-43e8-aba3-36b6350615ab","Type":"ContainerDied","Data":"b061d045043902bd8d96d8e080f5dad5f150c8c2ca6ed74ab812e818b62d53af"}
Oct 31 16:39:10.003036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.003007    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\" (UniqueName: \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\") pod \"d3399c25-2c2d-43e8-aba3-36b6350615ab\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") "
Oct 31 16:39:10.003394 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.003368    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\") pod \"d3399c25-2c2d-43e8-aba3-36b6350615ab\" (UID: \"d3399c25-2c2d-43e8-aba3-36b6350615ab\") "
Oct 31 16:39:10.003807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.003782    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume" pod "d3399c25-2c2d-43e8-aba3-36b6350615ab" (UID: "d3399c25-2c2d-43e8-aba3-36b6350615ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:10.004553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.004525    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmsfz\" (UniqueName: \"kubernetes.io/projected/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed-kube-api-access-jmsfz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:10.004689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.004559    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3399c25-2c2d-43e8-aba3-36b6350615ab-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:10.005072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.005049    4417 scope.go:122] "RemoveContainer" containerID="7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"
Oct 31 16:39:10.005407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.005387    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm" pod "d3399c25-2c2d-43e8-aba3-36b6350615ab" (UID: "d3399c25-2c2d-43e8-aba3-36b6350615ab"). InnerVolumeSpecName "kube-api-access-k62xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:10.005467 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:10.005407    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412\": container with ID starting with 7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412 not found: ID does not exist" containerID="7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"
Oct 31 16:39:10.005467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.005433    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412"} err="failed to get container status \"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412\": rpc error: code = NotFound desc = could not find container \"7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412\": container with ID starting with 7800f049c34f60dcc0906811812be64d4dfd938ef7e1de899155aad780cf7412 not found: ID does not exist"
Oct 31 16:39:10.005467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.005452    4417 scope.go:122] "RemoveContainer" containerID="6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"
Oct 31 16:39:10.016827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.016805    4417 scope.go:122] "RemoveContainer" containerID="6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"
Oct 31 16:39:10.017444 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:10.017417    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd\": container with ID starting with 6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd not found: ID does not exist" containerID="6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"
Oct 31 16:39:10.017560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.017453    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd"} err="failed to get container status \"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd\": rpc error: code = NotFound desc = could not find container \"6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd\": container with ID starting with 6d60552c514525dc63d6a661a8f70d74922f76b452dd4aff17ba8c67ee807ddd not found: ID does not exist"
Oct 31 16:39:10.105917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.105879    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62xm\" (UniqueName: \"kubernetes.io/projected/d3399c25-2c2d-43e8-aba3-36b6350615ab-kube-api-access-k62xm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:10.114699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.114569    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:39:10.118322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.118291    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1b30a15-77d5dd87f9-vzvhx"]
Oct 31 16:39:10.410710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.409690    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:39:10.413844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:10.413809    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1b30a15-65bcd6b779-pjxz2"]
Oct 31 16:39:11.052681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:11.052647    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" path="/var/lib/kubelet/pods/32b7bc81-0b35-46f1-aea3-1d8d417ee2ed/volumes"
Oct 31 16:39:11.053169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:11.053154    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3399c25-2c2d-43e8-aba3-36b6350615ab" path="/var/lib/kubelet/pods/d3399c25-2c2d-43e8-aba3-36b6350615ab/volumes"
Oct 31 16:39:11.812998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:11.812932    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" podUID="13ea2d49-7ec1-4778-9f81-c5793513f16b" containerName="payload-processing" containerID="cri-o://db0dde97385f4aa3e06bb669564831d6c1d8ad60a4e1b70744a0f3733911bb0b" gracePeriod=25
Oct 31 16:39:11.910717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:11.910637    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" podUID="b91f0df2-6f27-413f-ad30-ba80357cb953" containerName="payload-pre-processing" containerID="cri-o://ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195" gracePeriod=25
Oct 31 16:39:12.030319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.030256    4417 generic.go:356] "Generic (PLEG): container finished" podID="13ea2d49-7ec1-4778-9f81-c5793513f16b" containerID="db0dde97385f4aa3e06bb669564831d6c1d8ad60a4e1b70744a0f3733911bb0b" exitCode=0
Oct 31 16:39:12.030565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.030350    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" event={"ID":"13ea2d49-7ec1-4778-9f81-c5793513f16b","Type":"ContainerDied","Data":"db0dde97385f4aa3e06bb669564831d6c1d8ad60a4e1b70744a0f3733911bb0b"}
Oct 31 16:39:12.138146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.138109    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:39:12.267462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.256240    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\") pod \"13ea2d49-7ec1-4778-9f81-c5793513f16b\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") "
Oct 31 16:39:12.268521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.268344    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume" pod "13ea2d49-7ec1-4778-9f81-c5793513f16b" (UID: "13ea2d49-7ec1-4778-9f81-c5793513f16b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:12.269227 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.269199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\" (UniqueName: \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\") pod \"13ea2d49-7ec1-4778-9f81-c5793513f16b\" (UID: \"13ea2d49-7ec1-4778-9f81-c5793513f16b\") "
Oct 31 16:39:12.270965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.270942    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13ea2d49-7ec1-4778-9f81-c5793513f16b-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:12.296642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.276010    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs" pod "13ea2d49-7ec1-4778-9f81-c5793513f16b" (UID: "13ea2d49-7ec1-4778-9f81-c5793513f16b"). InnerVolumeSpecName "kube-api-access-mqcgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:12.383639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.382323    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcgs\" (UniqueName: \"kubernetes.io/projected/13ea2d49-7ec1-4778-9f81-c5793513f16b-kube-api-access-mqcgs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:12.405649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.401834    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:39:12.483172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.483133    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\" (UniqueName: \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\") pod \"b91f0df2-6f27-413f-ad30-ba80357cb953\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") "
Oct 31 16:39:12.483401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.483214    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\") pod \"b91f0df2-6f27-413f-ad30-ba80357cb953\" (UID: \"b91f0df2-6f27-413f-ad30-ba80357cb953\") "
Oct 31 16:39:12.483691 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.483664    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume" pod "b91f0df2-6f27-413f-ad30-ba80357cb953" (UID: "b91f0df2-6f27-413f-ad30-ba80357cb953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:12.485352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.485322    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq" pod "b91f0df2-6f27-413f-ad30-ba80357cb953" (UID: "b91f0df2-6f27-413f-ad30-ba80357cb953"). InnerVolumeSpecName "kube-api-access-lfplq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:12.585460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.585423    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfplq\" (UniqueName: \"kubernetes.io/projected/b91f0df2-6f27-413f-ad30-ba80357cb953-kube-api-access-lfplq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:12.585460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:12.585456    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91f0df2-6f27-413f-ad30-ba80357cb953-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:13.045868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.045830    4417 generic.go:356] "Generic (PLEG): container finished" podID="b91f0df2-6f27-413f-ad30-ba80357cb953" containerID="ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195" exitCode=0
Oct 31 16:39:13.046075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.046053    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"
Oct 31 16:39:13.052967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.052939    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"
Oct 31 16:39:13.059021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.058296    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" event={"ID":"b91f0df2-6f27-413f-ad30-ba80357cb953","Type":"ContainerDied","Data":"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"}
Oct 31 16:39:13.059021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.058356    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz" event={"ID":"b91f0df2-6f27-413f-ad30-ba80357cb953","Type":"ContainerDied","Data":"a0b761eb58ab61638da3b8c2aef178cb096cbc88280069d8104da62249f63396"}
Oct 31 16:39:13.059021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.058384    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb" event={"ID":"13ea2d49-7ec1-4778-9f81-c5793513f16b","Type":"ContainerDied","Data":"a203eeb69c9a5b257e3b4c51495ead2cf3c82afd732cffd1132ef46262e88d47"}
Oct 31 16:39:13.059021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.058416    4417 scope.go:122] "RemoveContainer" containerID="ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"
Oct 31 16:39:13.079525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.079496    4417 scope.go:122] "RemoveContainer" containerID="ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"
Oct 31 16:39:13.079883 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:13.079859    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195\": container with ID starting with ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195 not found: ID does not exist" containerID="ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"
Oct 31 16:39:13.079968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.079891    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195"} err="failed to get container status \"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195\": rpc error: code = NotFound desc = could not find container \"ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195\": container with ID starting with ebb7706a2c866ac00816e4406ed0e9547d7d2937016e4b04d9c2eaa83b102195 not found: ID does not exist"
Oct 31 16:39:13.079968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.079911    4417 scope.go:122] "RemoveContainer" containerID="db0dde97385f4aa3e06bb669564831d6c1d8ad60a4e1b70744a0f3733911bb0b"
Oct 31 16:39:13.230637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.228015    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:39:13.292648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.291411    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-636a7f-c7dd99447-zf9bz"]
Oct 31 16:39:13.326078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.326050    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:39:13.331280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.330434    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-636a7f-9859998f9-6fstb"]
Oct 31 16:39:13.581698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.577600    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:39:13.581698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.577867    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" containerID="cri-o://14e4322f0f4db42a39a33fbdedf6464f79f9b57d4b07d1b051be8bcdd58e2eac" gracePeriod=30
Oct 31 16:39:13.688195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.688159    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:39:13.688471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:13.688440    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy" containerID="cri-o://f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff" gracePeriod=30
Oct 31 16:39:15.055311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:15.055273    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ea2d49-7ec1-4778-9f81-c5793513f16b" path="/var/lib/kubelet/pods/13ea2d49-7ec1-4778-9f81-c5793513f16b/volumes"
Oct 31 16:39:15.055938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:15.055917    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91f0df2-6f27-413f-ad30-ba80357cb953" path="/var/lib/kubelet/pods/b91f0df2-6f27-413f-ad30-ba80357cb953/volumes"
Oct 31 16:39:16.525316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:16.525270    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sjzsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:39:16.525724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:16.525344    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:39:18.978257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:18.978218    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:39:19.149054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.149009    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.150401 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.150374    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.150648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.150629    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.166091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.166054    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.166472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.166451    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.167346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.166998    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:19.167904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.167884    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.168139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.168026    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.168299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.168279    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.168494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.168419    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.169013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.168993    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.181180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.179370    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.181180 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.180896    4417 generic.go:356] "Generic (PLEG): container finished" podID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerID="f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff" exitCode=0
Oct 31 16:39:19.181372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.180950    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" event={"ID":"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1","Type":"ContainerDied","Data":"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"}
Oct 31 16:39:19.181372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.181313    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s" event={"ID":"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1","Type":"ContainerDied","Data":"5a8e8cb0839a76da1526dce3695889d410ef53a3322b056d0b8b98af28dafcbb"}
Oct 31 16:39:19.181484 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.181371    4417 scope.go:122] "RemoveContainer" containerID="f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"
Oct 31 16:39:19.181540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.180997    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"
Oct 31 16:39:19.200936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.189787    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.200936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.193973    4417 generic.go:356] "Generic (PLEG): container finished" podID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerID="14e4322f0f4db42a39a33fbdedf6464f79f9b57d4b07d1b051be8bcdd58e2eac" exitCode=0
Oct 31 16:39:19.200936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.194199    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" event={"ID":"b95d0713-61d2-48be-9b5d-bb76bc0c4489","Type":"ContainerDied","Data":"14e4322f0f4db42a39a33fbdedf6464f79f9b57d4b07d1b051be8bcdd58e2eac"}
Oct 31 16:39:19.203716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.203668    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.204684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.203937    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\") pod \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\" (UID: \"dbcf44fe-5eeb-45e6-9806-7429c76cb2e1\") "
Oct 31 16:39:19.205634 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.205594    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.205774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.205757    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.205883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.205871    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.205967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.205957    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.206043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.206029    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.206131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.206119    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.207962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.207938    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.213368 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.213324    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "kube-api-access-9sddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.220356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.220304    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo" pod "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" (UID: "dbcf44fe-5eeb-45e6-9806-7429c76cb2e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:19.234883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.234743    4417 scope.go:122] "RemoveContainer" containerID="f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"
Oct 31 16:39:19.235396 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:19.235319    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff\": container with ID starting with f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff not found: ID does not exist" containerID="f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"
Oct 31 16:39:19.235396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.235358    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff"} err="failed to get container status \"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff\": rpc error: code = NotFound desc = could not find container \"f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff\": container with ID starting with f28a3a82a3894a8c5a4ee50eeb38b751a79eb407e013e5da280a7c4bbedc2cff not found: ID does not exist"
Oct 31 16:39:19.319863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.319802    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.319863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.319836    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sddg\" (UniqueName: \"kubernetes.io/projected/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-kube-api-access-9sddg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.319863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.319852    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.399659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.399631    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:39:19.522518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522481    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.522713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522570    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.522713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522662    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.522873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522736    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.522937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522871    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.522937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522907    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.523051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522973    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.523051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.522973    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:19.523225 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.523200    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.523299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.523255    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\") pod \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\" (UID: \"b95d0713-61d2-48be-9b5d-bb76bc0c4489\") "
Oct 31 16:39:19.523863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.523838    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.524113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.523202    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.524113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.524063    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.524290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.524136    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.524290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.524275    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.525852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.525825    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.526039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.526015    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:19.526526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.526501    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:39:19.526738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.526715    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5" pod "b95d0713-61d2-48be-9b5d-bb76bc0c4489" (UID: "b95d0713-61d2-48be-9b5d-bb76bc0c4489"). InnerVolumeSpecName "kube-api-access-2wbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:19.613830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.613788    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:39:19.617023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.616993    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-k2f8s"]
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626001    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626037    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626054    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626067    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626082    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbp5\" (UniqueName: \"kubernetes.io/projected/b95d0713-61d2-48be-9b5d-bb76bc0c4489-kube-api-access-2wbp5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626097    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626113    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:19.630459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:19.626126    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95d0713-61d2-48be-9b5d-bb76bc0c4489-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:20.206043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:20.206015    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"
Oct 31 16:39:20.206494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:20.206014    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr" event={"ID":"b95d0713-61d2-48be-9b5d-bb76bc0c4489","Type":"ContainerDied","Data":"5487fbd3148fd1d57a7550f6e63706c91bb3355614d03eea43c7ddde85864aee"}
Oct 31 16:39:20.206494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:20.206160    4417 scope.go:122] "RemoveContainer" containerID="14e4322f0f4db42a39a33fbdedf6464f79f9b57d4b07d1b051be8bcdd58e2eac"
Oct 31 16:39:20.363375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:20.363331    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:39:20.405999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:20.405968    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sjzsr"]
Oct 31 16:39:21.059640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:21.059306    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" path="/var/lib/kubelet/pods/b95d0713-61d2-48be-9b5d-bb76bc0c4489/volumes"
Oct 31 16:39:21.060660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:21.060021    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" path="/var/lib/kubelet/pods/dbcf44fe-5eeb-45e6-9806-7429c76cb2e1/volumes"
Oct 31 16:39:32.944889 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:32.944850    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:39:36.788486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:36.788442    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:39:39.546999 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.546962    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:39:39.548195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.547174    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" podUID="a5b53163-092a-440a-ae42-669f85d5ce5c" containerName="maas-api" containerID="cri-o://3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541" gracePeriod=30
Oct 31 16:39:39.877905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.877876    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:39:39.931315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.931281    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") pod \"a5b53163-092a-440a-ae42-669f85d5ce5c\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") "
Oct 31 16:39:39.931469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.931411    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\") pod \"a5b53163-092a-440a-ae42-669f85d5ce5c\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") "
Oct 31 16:39:39.931469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.931448    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\" (UniqueName: \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\") pod \"a5b53163-092a-440a-ae42-669f85d5ce5c\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") "
Oct 31 16:39:39.931553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.931542    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\") pod \"a5b53163-092a-440a-ae42-669f85d5ce5c\" (UID: \"a5b53163-092a-440a-ae42-669f85d5ce5c\") "
Oct 31 16:39:39.931811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.931785    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca" pod "a5b53163-092a-440a-ae42-669f85d5ce5c" (UID: "a5b53163-092a-440a-ae42-669f85d5ce5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:39.932107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.932086    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b53163-092a-440a-ae42-669f85d5ce5c-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:39.933606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.933579    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls" pod "a5b53163-092a-440a-ae42-669f85d5ce5c" (UID: "a5b53163-092a-440a-ae42-669f85d5ce5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:39.933810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.933788    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6" pod "a5b53163-092a-440a-ae42-669f85d5ce5c" (UID: "a5b53163-092a-440a-ae42-669f85d5ce5c"). InnerVolumeSpecName "kube-api-access-kzmb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:39.934071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:39.934053    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs" pod "a5b53163-092a-440a-ae42-669f85d5ce5c" (UID: "a5b53163-092a-440a-ae42-669f85d5ce5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:40.041661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.041606    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmb6\" (UniqueName: \"kubernetes.io/projected/a5b53163-092a-440a-ae42-669f85d5ce5c-kube-api-access-kzmb6\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:40.041661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.041663    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:40.041880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.041677    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b53163-092a-440a-ae42-669f85d5ce5c-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:40.352991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.352945    4417 generic.go:356] "Generic (PLEG): container finished" podID="a5b53163-092a-440a-ae42-669f85d5ce5c" containerID="3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541" exitCode=0
Oct 31 16:39:40.352991 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.352994    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"
Oct 31 16:39:40.353244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.353025    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" event={"ID":"a5b53163-092a-440a-ae42-669f85d5ce5c","Type":"ContainerDied","Data":"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"}
Oct 31 16:39:40.353244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.353095    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk" event={"ID":"a5b53163-092a-440a-ae42-669f85d5ce5c","Type":"ContainerDied","Data":"b55ddd1646f70fe42a8b2059ffc1b57b6c61130d4c69109b144ee29d14c5893c"}
Oct 31 16:39:40.353244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.353125    4417 scope.go:122] "RemoveContainer" containerID="3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"
Oct 31 16:39:40.365569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.365547    4417 scope.go:122] "RemoveContainer" containerID="3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"
Oct 31 16:39:40.365889 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:40.365865    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541\": container with ID starting with 3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541 not found: ID does not exist" containerID="3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"
Oct 31 16:39:40.365945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.365896    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541"} err="failed to get container status \"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541\": rpc error: code = NotFound desc = could not find container \"3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541\": container with ID starting with 3294052d9c5d94594f3cef4e008cca026c19405ce17fb4db0f322b08e9f7e541 not found: ID does not exist"
Oct 31 16:39:40.503919 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.503887    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:39:40.596046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:40.594583    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a1b30a15-68b644ff74-pzttk"]
Oct 31 16:39:41.050892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.050851    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b53163-092a-440a-ae42-669f85d5ce5c" path="/var/lib/kubelet/pods/a5b53163-092a-440a-ae42-669f85d5ce5c/volumes"
Oct 31 16:39:41.299418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.299382    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:39:41.300271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300251    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a5b53163-092a-440a-ae42-669f85d5ce5c" containerName="maas-api"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300273    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b53163-092a-440a-ae42-669f85d5ce5c" containerName="maas-api"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300302    4417 cpu_manager.go:414] "containerMap: removing container" podUID="61214a6c-c042-4c39-b547-429158e529df" containerName="maas-api"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300310    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61214a6c-c042-4c39-b547-429158e529df" containerName="maas-api"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300321    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d3399c25-2c2d-43e8-aba3-36b6350615ab" containerName="payload-pre-processing"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300328    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3399c25-2c2d-43e8-aba3-36b6350615ab" containerName="payload-pre-processing"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300344    4417 cpu_manager.go:414] "containerMap: removing container" podUID="13ea2d49-7ec1-4778-9f81-c5793513f16b" containerName="payload-processing"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300351    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ea2d49-7ec1-4778-9f81-c5793513f16b" containerName="payload-processing"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300368    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300375    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy"
Oct 31 16:39:41.300381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300385    4417 cpu_manager.go:414] "containerMap: removing container" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300392    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300415    4417 cpu_manager.go:414] "containerMap: removing container" podUID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" containerName="payload-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300423    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" containerName="payload-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300440    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b91f0df2-6f27-413f-ad30-ba80357cb953" containerName="payload-pre-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300447    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91f0df2-6f27-413f-ad30-ba80357cb953" containerName="payload-pre-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300472    4417 cpu_manager.go:414] "containerMap: removing container" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300479    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300636    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="61214a6c-c042-4c39-b547-429158e529df" containerName="maas-api"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300647    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3399c25-2c2d-43e8-aba3-36b6350615ab" containerName="payload-pre-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300656    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ea2d49-7ec1-4778-9f81-c5793513f16b" containerName="payload-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300668    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95d0713-61d2-48be-9b5d-bb76bc0c4489" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300678    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300688    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b7bc81-0b35-46f1-aea3-1d8d417ee2ed" containerName="payload-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300696    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91f0df2-6f27-413f-ad30-ba80357cb953" containerName="payload-pre-processing"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300707    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbcf44fe-5eeb-45e6-9806-7429c76cb2e1" containerName="istio-proxy"
Oct 31 16:39:41.300827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.300718    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b53163-092a-440a-ae42-669f85d5ce5c" containerName="maas-api"
Oct 31 16:39:41.306282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.306230    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.359842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.359809    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:39:41.402107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.402073    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-66db1c-serving-cert\""
Oct 31 16:39:41.407845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.407815    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v584v\" (UniqueName: \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.407977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.407939    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.408023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.407976    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.408023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.408015    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.509797 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.509758    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v584v\" (UniqueName: \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.510018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.509944    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.510106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.510040    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.510172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.510110    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.510372 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:41.510327    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-serving-cert: secret "maas-api-e2e-inf-b-66db1c-serving-cert" not found
Oct 31 16:39:41.510579 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:41.510563    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls podName:353b38eb-f642-4675-8809-bfc8d6ead71b nodeName:}" failed. No retries permitted until 2026-10-31 16:39:42.010539856 +0000 UTC m=+2227.072115035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls") pod "maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" (UID: "353b38eb-f642-4675-8809-bfc8d6ead71b") : secret "maas-api-e2e-inf-b-66db1c-serving-cert" not found
Oct 31 16:39:41.511061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.511035    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.514179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.514158    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:41.563087 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:41.563021    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v584v\" (UniqueName: \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:42.021055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:42.020969    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:42.023376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:42.023354    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") pod \"maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:42.262647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:42.257750    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:42.674896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:42.674870    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:39:42.677199 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:39:42.677173    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353b38eb_f642_4675_8809_bfc8d6ead71b.slice/crio-7912b59def6de1a749dc3a66d80665bb205734926e77a447ed9fd84985c32878 WatchSource:0}: Error finding container 7912b59def6de1a749dc3a66d80665bb205734926e77a447ed9fd84985c32878: Status 404 returned error can't find the container with id 7912b59def6de1a749dc3a66d80665bb205734926e77a447ed9fd84985c32878
Oct 31 16:39:43.373156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.373116    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" event={"ID":"353b38eb-f642-4675-8809-bfc8d6ead71b","Type":"ContainerStarted","Data":"7912b59def6de1a749dc3a66d80665bb205734926e77a447ed9fd84985c32878"}
Oct 31 16:39:43.807009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.806971    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:39:43.807774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.807738    4417 cpu_manager.go:414] "containerMap: removing container" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:43.807774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.807761    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:43.808033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.807960    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc3cb4c-9227-4c52-a024-b9564d7d06e4" containerName="istio-proxy"
Oct 31 16:39:43.811502 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.811480    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.868283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.868255    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:39:43.885641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885541    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.885811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885653    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.885811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885763    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.885931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885825    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.885931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885868    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdzc\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.885931 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885895    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.886108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.885972    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.886108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.886099    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.886230 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.886152    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.986816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986786    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986851    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986893    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdzc\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986914    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986937    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.986996    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987180    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987221    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987247    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987348    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987489    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987538    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.987600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.987561    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.988153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.988132    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.989395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.989372    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:43.990099 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:43.990062    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:44.040517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.040484    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:44.040923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.040900    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdzc\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ghk9c\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:44.125410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.125379    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:44.381376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.381332    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" event={"ID":"353b38eb-f642-4675-8809-bfc8d6ead71b","Type":"ContainerStarted","Data":"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"}
Oct 31 16:39:44.381776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.381407    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:44.489115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.489010    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" podStartSLOduration=2.549266536 podStartE2EDuration="3.488996347s" totalImagesPullingTime="939.729811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:44.485829857 +0000 UTC m=+2229.547405042" watchObservedRunningTime="2026-10-31 16:39:44.488996347 +0000 UTC m=+2229.550571532"
Oct 31 16:39:44.601223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.601182    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:39:44.602966 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:39:44.602937    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5900102_7c30_4633_b058_dc36fe25d45a.slice/crio-725e6d5ec5ce3c51faf2892511eec12b0ffeafeee0337eda1c4e9693520f869c WatchSource:0}: Error finding container 725e6d5ec5ce3c51faf2892511eec12b0ffeafeee0337eda1c4e9693520f869c: Status 404 returned error can't find the container with id 725e6d5ec5ce3c51faf2892511eec12b0ffeafeee0337eda1c4e9693520f869c
Oct 31 16:39:44.605497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.605458    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:44.605601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.605537    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:44.605601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:44.605585    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:45.390018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:45.389984    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" event={"ID":"c5900102-7c30-4633-b058-dc36fe25d45a","Type":"ContainerStarted","Data":"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"}
Oct 31 16:39:45.390789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:45.390037    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" event={"ID":"c5900102-7c30-4633-b058-dc36fe25d45a","Type":"ContainerStarted","Data":"725e6d5ec5ce3c51faf2892511eec12b0ffeafeee0337eda1c4e9693520f869c"}
Oct 31 16:39:45.501748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:45.501690    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podStartSLOduration=2.501675335 podStartE2EDuration="2.501675335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:45.500896112 +0000 UTC m=+2230.562471301" watchObservedRunningTime="2026-10-31 16:39:45.501675335 +0000 UTC m=+2230.563250520"
Oct 31 16:39:46.126532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:46.126495    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:46.128235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:46.128214    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:46.128321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:46.128257    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:47.127652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:47.126723    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:47.127652 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:47.126797    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:47.275423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:47.275385    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:39:48.125925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:48.125880    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:48.126141 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:48.125932    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:49.125710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:49.125678    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:49.126110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:49.125729    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:50.125750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:50.125710    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:50.126139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:50.125780    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:50.396413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:50.396325    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:39:51.126337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.126298    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:51.126863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.126367    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:51.258915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.258876    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:39:51.259177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.259138    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" podUID="22cd68ee-011b-4fda-b725-58000a5b6b23" containerName="maas-api" containerID="cri-o://126ffe6be43be06e564df7418be6f71b2c7fbc4690328854e557d30e5dc5e192" gracePeriod=30
Oct 31 16:39:51.430819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.430781    4417 generic.go:356] "Generic (PLEG): container finished" podID="22cd68ee-011b-4fda-b725-58000a5b6b23" containerID="126ffe6be43be06e564df7418be6f71b2c7fbc4690328854e557d30e5dc5e192" exitCode=0
Oct 31 16:39:51.431017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.430931    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" event={"ID":"22cd68ee-011b-4fda-b725-58000a5b6b23","Type":"ContainerDied","Data":"126ffe6be43be06e564df7418be6f71b2c7fbc4690328854e557d30e5dc5e192"}
Oct 31 16:39:51.612826 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.612794    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:39:51.742769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.742634    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\") pod \"22cd68ee-011b-4fda-b725-58000a5b6b23\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") "
Oct 31 16:39:51.742952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.742779    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") pod \"22cd68ee-011b-4fda-b725-58000a5b6b23\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") "
Oct 31 16:39:51.742952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.742824    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\" (UniqueName: \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\") pod \"22cd68ee-011b-4fda-b725-58000a5b6b23\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") "
Oct 31 16:39:51.743094 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.742953    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\") pod \"22cd68ee-011b-4fda-b725-58000a5b6b23\" (UID: \"22cd68ee-011b-4fda-b725-58000a5b6b23\") "
Oct 31 16:39:51.745944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.744985    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca" pod "22cd68ee-011b-4fda-b725-58000a5b6b23" (UID: "22cd68ee-011b-4fda-b725-58000a5b6b23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:39:51.746649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.746389    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs" pod "22cd68ee-011b-4fda-b725-58000a5b6b23" (UID: "22cd68ee-011b-4fda-b725-58000a5b6b23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:51.746974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.746828    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz" pod "22cd68ee-011b-4fda-b725-58000a5b6b23" (UID: "22cd68ee-011b-4fda-b725-58000a5b6b23"). InnerVolumeSpecName "kube-api-access-rtphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:51.748802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.748737    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls" pod "22cd68ee-011b-4fda-b725-58000a5b6b23" (UID: "22cd68ee-011b-4fda-b725-58000a5b6b23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:51.811126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.811078    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main" containerID="cri-o://84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2" gracePeriod=45
Oct 31 16:39:51.852677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.848850    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:51.852677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.848895    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22cd68ee-011b-4fda-b725-58000a5b6b23-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:51.852677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.848911    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22cd68ee-011b-4fda-b725-58000a5b6b23-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:51.852677 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:51.848925    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtphz\" (UniqueName: \"kubernetes.io/projected/22cd68ee-011b-4fda-b725-58000a5b6b23-kube-api-access-rtphz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.084569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.084538    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:39:52.126318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.126289    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ghk9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 16:39:52.126472 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.126337    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 16:39:52.160967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.160932    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.161137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161033    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.161137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161082    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\" (UniqueName: \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.161254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161228    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.161318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161252    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.161318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161289    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.161417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161359    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\") pod \"a09ada45-cd1f-4ca8-a326-373f37e106a2\" (UID: \"a09ada45-cd1f-4ca8-a326-373f37e106a2\") "
Oct 31 16:39:52.162520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.161735    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.162520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.162313    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.162520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.162411    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.162520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.162495    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.163564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.163542    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:39:52.164070 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.164051    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "kube-api-access-n22vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:39:52.164509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.164485    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs" pod "a09ada45-cd1f-4ca8-a326-373f37e106a2" (UID: "a09ada45-cd1f-4ca8-a326-373f37e106a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:39:52.279641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.273039    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09ada45-cd1f-4ca8-a326-373f37e106a2-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.279641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.273093    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.279641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.273109    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09ada45-cd1f-4ca8-a326-373f37e106a2-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.279641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.273124    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n22vh\" (UniqueName: \"kubernetes.io/projected/a09ada45-cd1f-4ca8-a326-373f37e106a2-kube-api-access-n22vh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.440154    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.440145    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65" event={"ID":"22cd68ee-011b-4fda-b725-58000a5b6b23","Type":"ContainerDied","Data":"691ba8fb8930b41f7f59773dcf0d4779da9bb3d3ea1a7d48d2347d024994ea00"}
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.440313    4417 scope.go:122] "RemoveContainer" containerID="126ffe6be43be06e564df7418be6f71b2c7fbc4690328854e557d30e5dc5e192"
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.442068    4417 generic.go:356] "Generic (PLEG): container finished" podID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerID="84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2" exitCode=0
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.442170    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" event={"ID":"a09ada45-cd1f-4ca8-a326-373f37e106a2","Type":"ContainerDied","Data":"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"}
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.442191    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9" event={"ID":"a09ada45-cd1f-4ca8-a326-373f37e106a2","Type":"ContainerDied","Data":"eb8b03b63382dd697b55d19b93c30d80e2ab85b75ce3534dd954547f035a932f"}
Oct 31 16:39:52.450701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.442232    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"
Oct 31 16:39:52.465246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.465225    4417 scope.go:122] "RemoveContainer" containerID="84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"
Oct 31 16:39:52.474965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.474948    4417 scope.go:122] "RemoveContainer" containerID="84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"
Oct 31 16:39:52.475222 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:39:52.475194    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2\": container with ID starting with 84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2 not found: ID does not exist" containerID="84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"
Oct 31 16:39:52.475317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.475233    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2"} err="failed to get container status \"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2\": rpc error: code = NotFound desc = could not find container \"84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2\": container with ID starting with 84ef677276b38cd265a1c79c80199984b4ef3f44d7f59cfdca01522f197ad6d2 not found: ID does not exist"
Oct 31 16:39:52.598680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.598642    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:39:52.602162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.602132    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636a7f-759b587b67-kph65"]
Oct 31 16:39:52.703366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.703278    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:39:52.707579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:52.707550    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/distinct-2-w1-4fe3a4-kserve-c9bdccdb4-kx6c9"]
Oct 31 16:39:53.054762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:53.054725    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cd68ee-011b-4fda-b725-58000a5b6b23" path="/var/lib/kubelet/pods/22cd68ee-011b-4fda-b725-58000a5b6b23/volumes"
Oct 31 16:39:53.055395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:53.055373    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" path="/var/lib/kubelet/pods/a09ada45-cd1f-4ca8-a326-373f37e106a2/volumes"
Oct 31 16:39:53.555176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:53.554787    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:53.556283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:53.556258    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:53.558071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:53.558052    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:39:58.927001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.926957    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:39:58.927773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927754    4417 cpu_manager.go:414] "containerMap: removing container" podUID="22cd68ee-011b-4fda-b725-58000a5b6b23" containerName="maas-api"
Oct 31 16:39:58.927773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927774    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cd68ee-011b-4fda-b725-58000a5b6b23" containerName="maas-api"
Oct 31 16:39:58.927890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927809    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main"
Oct 31 16:39:58.927890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927814    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main"
Oct 31 16:39:58.927953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927895    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a09ada45-cd1f-4ca8-a326-373f37e106a2" containerName="main"
Oct 31 16:39:58.927953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.927910    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cd68ee-011b-4fda-b725-58000a5b6b23" containerName="maas-api"
Oct 31 16:39:58.932882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:58.932857    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.010666 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.010602    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.010839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.010717    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.010839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.010764    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.010839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.010811    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.011028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.010948    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6qmq\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.011028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.011005    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.011144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.011068    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.011144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.011121    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.011251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.011159    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.033825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.033793    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:39:59.112022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.111981    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112029    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112138    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112205    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112299    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6qmq\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112347    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112397    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112442    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112477    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112755    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112755    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.112952 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112934    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.113016 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.112971    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.113077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.113052    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.114716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.114695    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.114823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.114807    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.164329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.164296    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.164643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.164602    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6qmq\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x2fs8\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.248069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.248006    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:39:59.668402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.668371    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:39:59.670209 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:39:59.670181    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201b9126_7885_426a_993a_07c4d4345ca5.slice/crio-4f955c9af59690288637ce058f4a3cb105b0ad282acb9801084f517bb9f1e8db WatchSource:0}: Error finding container 4f955c9af59690288637ce058f4a3cb105b0ad282acb9801084f517bb9f1e8db: Status 404 returned error can't find the container with id 4f955c9af59690288637ce058f4a3cb105b0ad282acb9801084f517bb9f1e8db
Oct 31 16:39:59.672859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.672825    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:59.672935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.672894    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:39:59.672935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:39:59.672923    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:40:00.514587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:00.514552    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" event={"ID":"201b9126-7885-426a-993a-07c4d4345ca5","Type":"ContainerStarted","Data":"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"}
Oct 31 16:40:00.515197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:00.515173    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" event={"ID":"201b9126-7885-426a-993a-07c4d4345ca5","Type":"ContainerStarted","Data":"4f955c9af59690288637ce058f4a3cb105b0ad282acb9801084f517bb9f1e8db"}
Oct 31 16:40:00.616239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:00.616206    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:40:00.616710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:00.616680    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" containerID="cri-o://5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc" gracePeriod=30
Oct 31 16:40:00.813647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:00.809111    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podStartSLOduration=2.809090181 podStartE2EDuration="2.809090181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:00.69052815 +0000 UTC m=+2245.752103338" watchObservedRunningTime="2026-10-31 16:40:00.809090181 +0000 UTC m=+2245.870665367"
Oct 31 16:40:01.252786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:01.252686    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:40:01.257789 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:01.257749    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:01.258095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:01.258064    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:02.256104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.256061    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:02.256756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.256721    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:02.296889 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.296857    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" podUID="1695e578-deae-4755-9b9f-92707ddb8275" containerName="main" containerID="cri-o://dc3d108618bb5d486841c8b0de6be7433766970f799e5cd65f4e8d622cb4e1d1" gracePeriod=45
Oct 31 16:40:02.355083 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.355041    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" podUID="1695e578-deae-4755-9b9f-92707ddb8275" containerName="main" probeResult="failure" output="Get \"https://10.133.0.134:8000/ready\": dial tcp 10.133.0.134:8000: connect: connection refused"
Oct 31 16:40:02.593248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.593117    4417 generic.go:356] "Generic (PLEG): container finished" podID="1695e578-deae-4755-9b9f-92707ddb8275" containerID="dc3d108618bb5d486841c8b0de6be7433766970f799e5cd65f4e8d622cb4e1d1" exitCode=0
Oct 31 16:40:02.593248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.593166    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" event={"ID":"1695e578-deae-4755-9b9f-92707ddb8275","Type":"ContainerDied","Data":"dc3d108618bb5d486841c8b0de6be7433766970f799e5cd65f4e8d622cb4e1d1"}
Oct 31 16:40:02.698508 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.698483    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:40:02.716808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.716781    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.716994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.716835    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\" (UniqueName: \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.716994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.716866    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\" (UniqueName: \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.716994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.716984    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.717172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717056    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.717172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717106    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\") pod \"1695e578-deae-4755-9b9f-92707ddb8275\" (UID: \"1695e578-deae-4755-9b9f-92707ddb8275\") "
Oct 31 16:40:02.717393 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717368    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:02.717533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717513    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:02.717636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717535    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:02.717839 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717824    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:02.717946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.717934    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:02.718051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.718040    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:02.720056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.720022    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:02.721658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.720342    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "kube-api-access-xx825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:02.721658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.720443    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm" pod "1695e578-deae-4755-9b9f-92707ddb8275" (UID: "1695e578-deae-4755-9b9f-92707ddb8275"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:02.819136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.819103    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx825\" (UniqueName: \"kubernetes.io/projected/1695e578-deae-4755-9b9f-92707ddb8275-kube-api-access-xx825\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:02.819136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.819134    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1695e578-deae-4755-9b9f-92707ddb8275-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:02.819136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:02.819144    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1695e578-deae-4755-9b9f-92707ddb8275-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:03.016175 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:03.016142    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:40:03.253643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.250088    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:03.253643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.250154    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:03.613636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.613528    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6" event={"ID":"1695e578-deae-4755-9b9f-92707ddb8275","Type":"ContainerDied","Data":"75bd6627b831071de8d0f0bce02b5652a923651f43ee2eb22dd3afb30dbc70dc"}
Oct 31 16:40:03.614332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.613665    4417 scope.go:122] "RemoveContainer" containerID="dc3d108618bb5d486841c8b0de6be7433766970f799e5cd65f4e8d622cb4e1d1"
Oct 31 16:40:03.614332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.613850    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"
Oct 31 16:40:03.787250 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.786874    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:40:03.790106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:03.790078    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a1b30a15/e2e-lifecycle-model-a1b30a15-kserve-8d677d6b9-f6kx6"]
Oct 31 16:40:04.250640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:04.250587    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:04.250842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:04.250669    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:04.525555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:04.525474    4417 patch_prober.go:29] interesting pod/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:40:04.525555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:04.525527    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:40:05.073833 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:05.073775    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1695e578-deae-4755-9b9f-92707ddb8275" path="/var/lib/kubelet/pods/1695e578-deae-4755-9b9f-92707ddb8275/volumes"
Oct 31 16:40:05.252588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:05.252102    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:05.252588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:05.252174    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:05.946706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:05.946677    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:40:06.099832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.099786    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.099857    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.099895    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100042    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100079    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100182    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100218    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100254    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.100312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100297    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\") pod \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\" (UID: \"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0\") "
Oct 31 16:40:06.101079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.100895    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:06.101231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.101095    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:06.101528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.101349    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:06.103659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.101682    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:06.103659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.103565    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:06.103659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.103590    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "kube-api-access-x2bj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:06.104237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.103905    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:06.106114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.106076    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:06.108890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.108858    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token" pod "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" (UID: "c9112b8c-a524-473e-8e2b-1a5c38f5e3a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202779    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202813    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202829    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202843    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202856    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202868    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202880    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bj4\" (UniqueName: \"kubernetes.io/projected/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-kube-api-access-x2bj4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202896    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.203005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.202908    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:06.249529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.249485    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:06.249770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.249554    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:06.637426 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.637394    4417 generic.go:356] "Generic (PLEG): container finished" podID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerID="5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc" exitCode=0
Oct 31 16:40:06.637630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.637509    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerDied","Data":"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"}
Oct 31 16:40:06.637630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.637549    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8" event={"ID":"c9112b8c-a524-473e-8e2b-1a5c38f5e3a0","Type":"ContainerDied","Data":"43d74d43e5bb5f80603e6813b60ad3bcaab2a24ce9cc4ae9b3abad8f2daab4cf"}
Oct 31 16:40:06.637630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.637570    4417 scope.go:122] "RemoveContainer" containerID="5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"
Oct 31 16:40:06.637830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.637710    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"
Oct 31 16:40:06.656717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.656694    4417 scope.go:122] "RemoveContainer" containerID="33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"
Oct 31 16:40:06.669766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.669730    4417 scope.go:122] "RemoveContainer" containerID="5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"
Oct 31 16:40:06.670127 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:06.670103    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc\": container with ID starting with 5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc not found: ID does not exist" containerID="5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"
Oct 31 16:40:06.670248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.670136    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc"} err="failed to get container status \"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc\": rpc error: code = NotFound desc = could not find container \"5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc\": container with ID starting with 5bfe1305478b217513263ca573f556c6471e3b1713e84034f81b9be04cd978cc not found: ID does not exist"
Oct 31 16:40:06.670248 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.670162    4417 scope.go:122] "RemoveContainer" containerID="33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"
Oct 31 16:40:06.670526 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:06.670496    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e\": container with ID starting with 33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e not found: ID does not exist" containerID="33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"
Oct 31 16:40:06.670723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.670685    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e"} err="failed to get container status \"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e\": rpc error: code = NotFound desc = could not find container \"33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e\": container with ID starting with 33e7bc32960f3ef67bf683592e5705bd3eb7ff39f588d061425d2ba7ec39194e not found: ID does not exist"
Oct 31 16:40:06.792753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.792709    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:40:06.796768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:06.796738    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-636a7f-openshift-default-cc597bf49-fgmh8"]
Oct 31 16:40:07.055058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:07.055023    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" path="/var/lib/kubelet/pods/c9112b8c-a524-473e-8e2b-1a5c38f5e3a0/volumes"
Oct 31 16:40:07.248658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:07.248590    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x2fs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 16:40:07.249273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:07.249236    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 16:40:08.509941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:08.509907    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:40:08.511537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:08.511514    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:40:08.512288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:08.512262    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:40:08.940890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:08.939853    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": read tcp 10.133.0.2:45490->10.133.0.162:15021: read: connection reset by peer" start-of-body=
Oct 31 16:40:08.940890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:08.939920    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": read tcp 10.133.0.2:45490->10.133.0.162:15021: read: connection reset by peer"
Oct 31 16:40:09.662234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.662202    4417 generic.go:356] "Generic (PLEG): container finished" podID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerID="15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8" exitCode=0
Oct 31 16:40:09.662687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.662283    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerDied","Data":"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"}
Oct 31 16:40:09.662880 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.662860    4417 scope.go:122] "RemoveContainer" containerID="15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"
Oct 31 16:40:09.664562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.664528    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:40:09.664657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.664632    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:40:09.664710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:09.664689    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:40:10.670670 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:10.670607    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerStarted","Data":"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"}
Oct 31 16:40:11.594904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:11.594860    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:40:11.595089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:11.595065    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 16:40:11.595158 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:11.595119    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 16:40:12.595147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:12.595109    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 16:40:12.595558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:12.595165    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 16:40:13.597403 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:13.597358    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 16:40:13.598179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:13.598146    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 16:40:14.595560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:14.595521    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 16:40:14.595774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:14.595587    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 16:40:15.598287 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:15.598243    4417 patch_prober.go:29] interesting pod/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 31 16:40:15.598834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:15.598320    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 31 16:40:16.599453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:16.599422    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:40:16.599867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:16.599845    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:40:16.600500 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:16.600477    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:40:25.302722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:25.302685    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:40:25.348373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:25.348344    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:40:32.278170 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:32.278136    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:40:32.378967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:32.378934    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:40:32.944604 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:32.944572    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:40:37.278462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:37.278413    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing" containerID="cri-o://fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0" gracePeriod=25
Oct 31 16:40:37.379707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:37.379604    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing" containerID="cri-o://6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d" gracePeriod=25
Oct 31 16:40:40.324573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.324536    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main" containerID="cri-o://db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b" gracePeriod=45
Oct 31 16:40:40.368202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.368171    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main" containerID="cri-o://a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10" gracePeriod=45
Oct 31 16:40:40.756818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.756784    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:40:40.790065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790039    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790101    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790218    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\" (UniqueName: \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790278 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790261    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790370    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790400    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.790513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790460    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\") pod \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\" (UID: \"c9fccb4f-41d9-4f92-b721-ecd3694120c6\") "
Oct 31 16:40:40.790773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790748    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.790907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.790880    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.791036 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.791019    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.791096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.791041    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.791096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.791056    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.792932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.792904    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "kube-api-access-k2t8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:40.793148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.793115    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.794026 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.793998    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs" pod "c9fccb4f-41d9-4f92-b721-ecd3694120c6" (UID: "c9fccb4f-41d9-4f92-b721-ecd3694120c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:40.801911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.801891    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:40:40.891967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.891879    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.892139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.891982    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.892139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892020    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.892139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892099    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.892139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892134    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.892387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892202    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\" (UniqueName: \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\") pod \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\" (UID: \"cfd98130-d605-4f94-8ba5-eb8afa6a8390\") "
Oct 31 16:40:40.893167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892791    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2t8b\" (UniqueName: \"kubernetes.io/projected/c9fccb4f-41d9-4f92-b721-ecd3694120c6-kube-api-access-k2t8b\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.893167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892822    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fccb4f-41d9-4f92-b721-ecd3694120c6-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.893167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892837    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fccb4f-41d9-4f92-b721-ecd3694120c6-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.893167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.892853    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.893521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.893325    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.893707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.893672    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.895575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.895548    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "kube-api-access-nk84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:40.899747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.899724    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:40.899860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.899744    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs" pod "cfd98130-d605-4f94-8ba5-eb8afa6a8390" (UID: "cfd98130-d605-4f94-8ba5-eb8afa6a8390"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:40:40.920697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.920675    4417 generic.go:356] "Generic (PLEG): container finished" podID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerID="db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b" exitCode=0
Oct 31 16:40:40.920808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.920714    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"
Oct 31 16:40:40.920808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.920755    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" event={"ID":"c9fccb4f-41d9-4f92-b721-ecd3694120c6","Type":"ContainerDied","Data":"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"}
Oct 31 16:40:40.920808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.920796    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs" event={"ID":"c9fccb4f-41d9-4f92-b721-ecd3694120c6","Type":"ContainerDied","Data":"5f8c3c17ba77d0664f74d00fbd984ee257f6906e8d836ae1a410fbb9ea4ae3dd"}
Oct 31 16:40:40.920997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.920815    4417 scope.go:122] "RemoveContainer" containerID="db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"
Oct 31 16:40:40.922306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.922285    4417 generic.go:356] "Generic (PLEG): container finished" podID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerID="a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10" exitCode=0
Oct 31 16:40:40.922413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.922317    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"
Oct 31 16:40:40.922413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.922340    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" event={"ID":"cfd98130-d605-4f94-8ba5-eb8afa6a8390","Type":"ContainerDied","Data":"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"}
Oct 31 16:40:40.922413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.922368    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6" event={"ID":"cfd98130-d605-4f94-8ba5-eb8afa6a8390","Type":"ContainerDied","Data":"3ad527d473d0f07c7f8c9c10b7709a21adf411ce8bf7a6bdacb2a560526860d7"}
Oct 31 16:40:40.932156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.932134    4417 scope.go:122] "RemoveContainer" containerID="db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"
Oct 31 16:40:40.932442 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:40.932417    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b\": container with ID starting with db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b not found: ID does not exist" containerID="db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"
Oct 31 16:40:40.932529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.932454    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b"} err="failed to get container status \"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b\": rpc error: code = NotFound desc = could not find container \"db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b\": container with ID starting with db7c7195ccbdb4bf8d8d2cbb51a0c8eb449fa2bc86fce5d9c43e345a2bca732b not found: ID does not exist"
Oct 31 16:40:40.932529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.932476    4417 scope.go:122] "RemoveContainer" containerID="a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"
Oct 31 16:40:40.944338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.944321    4417 scope.go:122] "RemoveContainer" containerID="a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"
Oct 31 16:40:40.944641 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:40.944599    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10\": container with ID starting with a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10 not found: ID does not exist" containerID="a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"
Oct 31 16:40:40.944708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.944640    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10"} err="failed to get container status \"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10\": rpc error: code = NotFound desc = could not find container \"a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10\": container with ID starting with a7117bae2806672d9dad6ced88f6d9c1ca0413df188dfaca2136336054ffef10 not found: ID does not exist"
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993913    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993945    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993961    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993973    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993983    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd98130-d605-4f94-8ba5-eb8afa6a8390-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:40.998522 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:40.993999    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk84t\" (UniqueName: \"kubernetes.io/projected/cfd98130-d605-4f94-8ba5-eb8afa6a8390-kube-api-access-nk84t\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:41.077045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:41.077013    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:40:41.081020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:41.080981    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/premium-w4-59357a-kserve-5bbb8d4f6-mlsgs"]
Oct 31 16:40:41.209697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:41.207068    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:40:41.213397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:41.213337    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/simulator-w4-59357a-kserve-849d764dc7-4f2p6"]
Oct 31 16:40:43.106639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:43.105370    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" path="/var/lib/kubelet/pods/c9fccb4f-41d9-4f92-b721-ecd3694120c6/volumes"
Oct 31 16:40:43.106639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:43.106079    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" path="/var/lib/kubelet/pods/cfd98130-d605-4f94-8ba5-eb8afa6a8390/volumes"
Oct 31 16:40:43.570191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:43.570152    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:40:43.570741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:43.570680    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy" containerID="cri-o://f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d" gracePeriod=30
Oct 31 16:40:44.911471 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.911429    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:40:44.912060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.911744    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy" containerID="cri-o://e2d9c19bf53fe878308183686c059b23d7246c983fb6b88912aa17303f8a0bf1" gracePeriod=30
Oct 31 16:40:44.946346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.946318    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:40:44.946531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.946510    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" containerID="cri-o://577486bbd4113722bee1a54c4bf74c5b8ba865951f432df52e3750b936a1bd7c" gracePeriod=30
Oct 31 16:40:44.951582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.951552    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:40:44.951842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.951801    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" containerID="cri-o://5e1c74bbc2325d7e3010ec4676d9dd7c3ada844f9f54742196f8d81d3f3a7dad" gracePeriod=30
Oct 31 16:40:44.956604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.956582    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:40:44.956786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:44.956768    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy" containerID="cri-o://144d71ab8854ce3c619494dbd7fb733aea992618f812cbf8190dc0726c17b7bc" gracePeriod=30
Oct 31 16:40:45.729608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:45.729573    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:40:45.729795 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:45.729573    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:40:45.737381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:45.737357    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:40:45.737488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:45.737395    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:40:46.574965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:46.574921    4417 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:40:46.575347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:46.574984    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:40:48.843844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.843783    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:40:48.917311 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917284    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917383    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917452    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917499    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917534    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917563    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917578    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917575    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917657    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.917711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917708    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\") pod \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\" (UID: \"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4\") "
Oct 31 16:40:48.918114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917817    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:48.918114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.917927    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:48.918193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918110    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:48.918330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918306    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:48.918422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918336    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:48.918422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918354    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:48.918422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918367    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:48.918422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.918381    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:48.919823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.919800    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:48.919941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.919919    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:48.919941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.919926    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:48.920119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:48.920102    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx" pod "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" (UID: "72307cd0-e3bf-48df-b4eb-890b9d3a4ad4"). InnerVolumeSpecName "kube-api-access-f4brx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:49.020337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.020304    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.020337 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.020341    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.020526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.020356    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.020526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.020372    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4brx\" (UniqueName: \"kubernetes.io/projected/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4-kube-api-access-f4brx\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:49.075955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.068487    4417 generic.go:356] "Generic (PLEG): container finished" podID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerID="f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d" exitCode=0
Oct 31 16:40:49.075955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.068529    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"
Oct 31 16:40:49.075955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.068559    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" event={"ID":"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4","Type":"ContainerDied","Data":"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"}
Oct 31 16:40:49.075955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.068646    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2" event={"ID":"72307cd0-e3bf-48df-b4eb-890b9d3a4ad4","Type":"ContainerDied","Data":"518327e1c4ddfd84bef96c24caaee2c9723c3ca76d2e3b1ef74ca681ab17b9de"}
Oct 31 16:40:49.075955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.068678    4417 scope.go:122] "RemoveContainer" containerID="f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"
Oct 31 16:40:49.080246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.080226    4417 scope.go:122] "RemoveContainer" containerID="f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"
Oct 31 16:40:49.080534 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:49.080512    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d\": container with ID starting with f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d not found: ID does not exist" containerID="f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"
Oct 31 16:40:49.080594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.080541    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d"} err="failed to get container status \"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d\": rpc error: code = NotFound desc = could not find container \"f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d\": container with ID starting with f94aca6df23cfe87f86bba38edc0511ca799017c3a9d082767461a6829a5a40d not found: ID does not exist"
Oct 31 16:40:49.220791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.220702    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:40:49.223705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:49.223684    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-vnmp2"]
Oct 31 16:40:50.079935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.079890    4417 generic.go:356] "Generic (PLEG): container finished" podID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerID="577486bbd4113722bee1a54c4bf74c5b8ba865951f432df52e3750b936a1bd7c" exitCode=0
Oct 31 16:40:50.080317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.080078    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" event={"ID":"6ab92634-4b15-409c-abb0-c45ff14e38f8","Type":"ContainerDied","Data":"577486bbd4113722bee1a54c4bf74c5b8ba865951f432df52e3750b936a1bd7c"}
Oct 31 16:40:50.084285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.084255    4417 generic.go:356] "Generic (PLEG): container finished" podID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerID="144d71ab8854ce3c619494dbd7fb733aea992618f812cbf8190dc0726c17b7bc" exitCode=0
Oct 31 16:40:50.084387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.084365    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" event={"ID":"6f30e01f-316d-42db-a49f-e76808bf20fe","Type":"ContainerDied","Data":"144d71ab8854ce3c619494dbd7fb733aea992618f812cbf8190dc0726c17b7bc"}
Oct 31 16:40:50.086047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.086023    4417 generic.go:356] "Generic (PLEG): container finished" podID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerID="e2d9c19bf53fe878308183686c059b23d7246c983fb6b88912aa17303f8a0bf1" exitCode=0
Oct 31 16:40:50.086165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.086082    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" event={"ID":"9917d9f7-bdc6-4be5-a008-d0d8ad96254d","Type":"ContainerDied","Data":"e2d9c19bf53fe878308183686c059b23d7246c983fb6b88912aa17303f8a0bf1"}
Oct 31 16:40:50.088711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.088685    4417 generic.go:356] "Generic (PLEG): container finished" podID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerID="5e1c74bbc2325d7e3010ec4676d9dd7c3ada844f9f54742196f8d81d3f3a7dad" exitCode=0
Oct 31 16:40:50.088787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.088730    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerDied","Data":"5e1c74bbc2325d7e3010ec4676d9dd7c3ada844f9f54742196f8d81d3f3a7dad"}
Oct 31 16:40:50.088787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.088776    4417 scope.go:122] "RemoveContainer" containerID="4dca61d2db839764a81fbdc2d93aca2dd32309850535195b121f227d0c1a2817"
Oct 31 16:40:50.235791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.235764    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:40:50.260560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.260533    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:40:50.309630 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.309593    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:40:50.324738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.324720    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:40:50.350146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350122    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.350264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350188    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.350264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350234    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350285    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350317    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350368    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350430    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350460    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350477    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350489    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350526    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350554    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350579    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350732    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350764    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350772    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350834    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350870    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\") pod \"6ab92634-4b15-409c-abb0-c45ff14e38f8\" (UID: \"6ab92634-4b15-409c-abb0-c45ff14e38f8\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350956    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350983    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\") pod \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\" (UID: \"9917d9f7-bdc6-4be5-a008-d0d8ad96254d\") "
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350984    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.350999    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351019    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351383    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351715    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351739    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351755    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351768    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351781    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab92634-4b15-409c-abb0-c45ff14e38f8-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351794    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.351991    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.352794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.352134    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.355176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.352818    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.355176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.352844    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.355176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.354425    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.362200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362135    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.362299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362223    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.362299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362244    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.362478 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362398    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k" pod "6ab92634-4b15-409c-abb0-c45ff14e38f8" (UID: "6ab92634-4b15-409c-abb0-c45ff14e38f8"). InnerVolumeSpecName "kube-api-access-86l6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.363117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362659    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "kube-api-access-vpvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.363117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362733    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.363117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.362936    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo" pod "9917d9f7-bdc6-4be5-a008-d0d8ad96254d" (UID: "9917d9f7-bdc6-4be5-a008-d0d8ad96254d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.452414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452387    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452446    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452477    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452503    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452521    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452548    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452588    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452663    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452671    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452708    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452756    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452799    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452848    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452851    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452873    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452927    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452936    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.452950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452951    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.452987    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453064    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453117    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\") pod \"6f30e01f-316d-42db-a49f-e76808bf20fe\" (UID: \"6f30e01f-316d-42db-a49f-e76808bf20fe\") "
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453143    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\") pod \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\" (UID: \"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8\") "
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453226    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.453553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453372    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.453881 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453576    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.453934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453902    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:50.453934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453916    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453943    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453961    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453977    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86l6k\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-kube-api-access-86l6k\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.453992    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454005    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454019    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454032    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454045    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454067    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454080    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454094    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab92634-4b15-409c-abb0-c45ff14e38f8-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454107    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454120    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454136    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454149    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454162    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvsp\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-kube-api-access-vpvsp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454180    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454193    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917d9f7-bdc6-4be5-a008-d0d8ad96254d-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.454388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454387    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.455035 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.454556    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.455708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.455674    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.455905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.455690    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "kube-api-access-788g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.455905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.455823    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:40:50.457357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.457324    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.457455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.457367    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token" pod "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" (UID: "5e3b69c1-72fb-45f5-81f3-4cceef64c2f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.457455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.457393    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "kube-api-access-x5n8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.458530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.458507    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:40:50.458643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.458539    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token" pod "6f30e01f-316d-42db-a49f-e76808bf20fe" (UID: "6f30e01f-316d-42db-a49f-e76808bf20fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:50.555932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555904    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f30e01f-316d-42db-a49f-e76808bf20fe-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.555932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555929    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.555932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555939    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555948    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555960    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555969    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5n8v\" (UniqueName: \"kubernetes.io/projected/6f30e01f-316d-42db-a49f-e76808bf20fe-kube-api-access-x5n8v\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555978    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555985    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.555994    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788g7\" (UniqueName: \"kubernetes.io/projected/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-kube-api-access-788g7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.556002    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:50.556169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:50.556012    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f30e01f-316d-42db-a49f-e76808bf20fe-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:51.051281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.051237    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" path="/var/lib/kubelet/pods/72307cd0-e3bf-48df-b4eb-890b9d3a4ad4/volumes"
Oct 31 16:40:51.095675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.095652    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"
Oct 31 16:40:51.096110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.095648    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8" event={"ID":"6f30e01f-316d-42db-a49f-e76808bf20fe","Type":"ContainerDied","Data":"90876a7fefab6bdfde836640f3aff88e85770e9aebc58830da8b549e9dd52d31"}
Oct 31 16:40:51.096110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.095789    4417 scope.go:122] "RemoveContainer" containerID="144d71ab8854ce3c619494dbd7fb733aea992618f812cbf8190dc0726c17b7bc"
Oct 31 16:40:51.099033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.097371    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh" event={"ID":"9917d9f7-bdc6-4be5-a008-d0d8ad96254d","Type":"ContainerDied","Data":"b7dfc75fd4e9173568014aa0e7510a7c520a4078260fa0350f3a35ca8fe6f480"}
Oct 31 16:40:51.099033 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.097427    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"
Oct 31 16:40:51.099410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.099393    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"
Oct 31 16:40:51.099410 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.099402    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j" event={"ID":"5e3b69c1-72fb-45f5-81f3-4cceef64c2f8","Type":"ContainerDied","Data":"98178f1c2d6c352882b07827035b8596ed468582c4ac02c63da112233b2eff77"}
Oct 31 16:40:51.100995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.100977    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz" event={"ID":"6ab92634-4b15-409c-abb0-c45ff14e38f8","Type":"ContainerDied","Data":"303b69a0dbf26466fb4878b4314f4e06549b49b494e9e4d2a691f8819462627c"}
Oct 31 16:40:51.101108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.101002    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"
Oct 31 16:40:51.110123 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.110101    4417 scope.go:122] "RemoveContainer" containerID="e2d9c19bf53fe878308183686c059b23d7246c983fb6b88912aa17303f8a0bf1"
Oct 31 16:40:51.118425 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.118405    4417 scope.go:122] "RemoveContainer" containerID="5e1c74bbc2325d7e3010ec4676d9dd7c3ada844f9f54742196f8d81d3f3a7dad"
Oct 31 16:40:51.126769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.126742    4417 scope.go:122] "RemoveContainer" containerID="577486bbd4113722bee1a54c4bf74c5b8ba865951f432df52e3750b936a1bd7c"
Oct 31 16:40:51.247724 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.247634    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:40:51.252629 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.252582    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-ht7mz"]
Oct 31 16:40:51.351934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.351685    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:40:51.354853 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.354822    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-gvzj8"]
Oct 31 16:40:51.454865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.454827    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:40:51.461387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.461357    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-dxd7j"]
Oct 31 16:40:51.563546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.563511    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:40:51.567386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:51.567359    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-khjnh"]
Oct 31 16:40:53.049900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:53.049866    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" path="/var/lib/kubelet/pods/5e3b69c1-72fb-45f5-81f3-4cceef64c2f8/volumes"
Oct 31 16:40:53.050392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:53.050377    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" path="/var/lib/kubelet/pods/6ab92634-4b15-409c-abb0-c45ff14e38f8/volumes"
Oct 31 16:40:53.050816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:53.050803    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" path="/var/lib/kubelet/pods/6f30e01f-316d-42db-a49f-e76808bf20fe/volumes"
Oct 31 16:40:53.051531 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:53.051518    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" path="/var/lib/kubelet/pods/9917d9f7-bdc6-4be5-a008-d0d8ad96254d/volumes"
Oct 31 16:40:55.729468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:55.729435    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:40:55.729944 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:55.729482    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:40:57.561397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.561374    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:40:57.632783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.632751    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\") pod \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") "
Oct 31 16:40:57.632973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.632846    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\" (UniqueName: \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\") pod \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\" (UID: \"2dd810ac-b67c-4162-b82f-fc1ccc94ca17\") "
Oct 31 16:40:57.633174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.633150    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume" pod "2dd810ac-b67c-4162-b82f-fc1ccc94ca17" (UID: "2dd810ac-b67c-4162-b82f-fc1ccc94ca17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:57.636700 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.636675    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq" pod "2dd810ac-b67c-4162-b82f-fc1ccc94ca17" (UID: "2dd810ac-b67c-4162-b82f-fc1ccc94ca17"). InnerVolumeSpecName "kube-api-access-jtshq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:57.661008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.660986    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:40:57.733830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.733803    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\") pod \"20dde006-e3d8-46ed-bb12-9619efbd4414\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") "
Oct 31 16:40:57.734015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.734000    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\" (UniqueName: \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\") pod \"20dde006-e3d8-46ed-bb12-9619efbd4414\" (UID: \"20dde006-e3d8-46ed-bb12-9619efbd4414\") "
Oct 31 16:40:57.734177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.734132    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume" pod "20dde006-e3d8-46ed-bb12-9619efbd4414" (UID: "20dde006-e3d8-46ed-bb12-9619efbd4414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:40:57.734378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.734365    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20dde006-e3d8-46ed-bb12-9619efbd4414-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:57.734415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.734381    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:57.734415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.734390    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtshq\" (UniqueName: \"kubernetes.io/projected/2dd810ac-b67c-4162-b82f-fc1ccc94ca17-kube-api-access-jtshq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:57.736042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.736025    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz" pod "20dde006-e3d8-46ed-bb12-9619efbd4414" (UID: "20dde006-e3d8-46ed-bb12-9619efbd4414"). InnerVolumeSpecName "kube-api-access-mpclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:40:57.835186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:57.835155    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpclz\" (UniqueName: \"kubernetes.io/projected/20dde006-e3d8-46ed-bb12-9619efbd4414-kube-api-access-mpclz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:40:58.145602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.145510    4417 generic.go:356] "Generic (PLEG): container finished" podID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerID="fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0" exitCode=0
Oct 31 16:40:58.145602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.145562    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"
Oct 31 16:40:58.145602 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.145591    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" event={"ID":"2dd810ac-b67c-4162-b82f-fc1ccc94ca17","Type":"ContainerDied","Data":"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"}
Oct 31 16:40:58.145941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.145669    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls" event={"ID":"2dd810ac-b67c-4162-b82f-fc1ccc94ca17","Type":"ContainerDied","Data":"80add3f4491c38db858c56258cfea68228b14d57ed2399c60752a6bb9708b5ce"}
Oct 31 16:40:58.145941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.145698    4417 scope.go:122] "RemoveContainer" containerID="fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"
Oct 31 16:40:58.147077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.147038    4417 generic.go:356] "Generic (PLEG): container finished" podID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerID="6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d" exitCode=0
Oct 31 16:40:58.147207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.147078    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" event={"ID":"20dde006-e3d8-46ed-bb12-9619efbd4414","Type":"ContainerDied","Data":"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"}
Oct 31 16:40:58.147207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.147096    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"
Oct 31 16:40:58.147207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.147113    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j" event={"ID":"20dde006-e3d8-46ed-bb12-9619efbd4414","Type":"ContainerDied","Data":"da9789b580a17b91259fbdb343dddf36d37c142ac344cce2913d398f0cc85aff"}
Oct 31 16:40:58.155979 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.155807    4417 scope.go:122] "RemoveContainer" containerID="fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"
Oct 31 16:40:58.156188 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:58.156156    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0\": container with ID starting with fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0 not found: ID does not exist" containerID="fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"
Oct 31 16:40:58.156310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.156191    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0"} err="failed to get container status \"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0\": rpc error: code = NotFound desc = could not find container \"fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0\": container with ID starting with fbdd9ecea41817383ef2d0e99c6f42f3dfd95429a605ed8a475160928dc6d1b0 not found: ID does not exist"
Oct 31 16:40:58.156310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.156215    4417 scope.go:122] "RemoveContainer" containerID="6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"
Oct 31 16:40:58.165624 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.165598    4417 scope.go:122] "RemoveContainer" containerID="6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"
Oct 31 16:40:58.165903 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:40:58.165883    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d\": container with ID starting with 6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d not found: ID does not exist" containerID="6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"
Oct 31 16:40:58.165981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.165909    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d"} err="failed to get container status \"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d\": rpc error: code = NotFound desc = could not find container \"6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d\": container with ID starting with 6a8e1bc67f89ec301090d950437fc43372a5eaaaa76b49eb98d60d4e6474ed8d not found: ID does not exist"
Oct 31 16:40:58.297236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.297210    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:40:58.301563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.301537    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-59357a-88f6c5975-dm79j"]
Oct 31 16:40:58.402112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.402031    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:40:58.405195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:58.405170    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-59357a-7df684b4b7-wbbls"]
Oct 31 16:40:59.050469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.050437    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" path="/var/lib/kubelet/pods/20dde006-e3d8-46ed-bb12-9619efbd4414/volumes"
Oct 31 16:40:59.050852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.050836    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" path="/var/lib/kubelet/pods/2dd810ac-b67c-4162-b82f-fc1ccc94ca17/volumes"
Oct 31 16:40:59.408768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.408683    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:40:59.409200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409187    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main"
Oct 31 16:40:59.409245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409201    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main"
Oct 31 16:40:59.409245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409217    4417 cpu_manager.go:414] "containerMap: removing container" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing"
Oct 31 16:40:59.409245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409222    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing"
Oct 31 16:40:59.409245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409233    4417 cpu_manager.go:414] "containerMap: removing container" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy"
Oct 31 16:40:59.409245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409238    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409248    4417 cpu_manager.go:414] "containerMap: removing container" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409253    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409263    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409268    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409279    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409284    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409292    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1695e578-deae-4755-9b9f-92707ddb8275" containerName="main"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409296    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1695e578-deae-4755-9b9f-92707ddb8275" containerName="main"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409303    4417 cpu_manager.go:414] "containerMap: removing container" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409308    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409319    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409323    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409332    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409337    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409343    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409348    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409356    4417 cpu_manager.go:414] "containerMap: removing container" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409360    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409367    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy"
Oct 31 16:40:59.409396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409372    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409440    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dde006-e3d8-46ed-bb12-9619efbd4414" containerName="payload-pre-processing"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409448    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="72307cd0-e3bf-48df-b4eb-890b9d3a4ad4" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409454    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd810ac-b67c-4162-b82f-fc1ccc94ca17" containerName="payload-processing"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409460    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9917d9f7-bdc6-4be5-a008-d0d8ad96254d" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409468    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409474    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1695e578-deae-4755-9b9f-92707ddb8275" containerName="main"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409479    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409484    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab92634-4b15-409c-abb0-c45ff14e38f8" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409489    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9112b8c-a524-473e-8e2b-1a5c38f5e3a0" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409494    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fccb4f-41d9-4f92-b721-ecd3694120c6" containerName="main"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409500    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f30e01f-316d-42db-a49f-e76808bf20fe" containerName="istio-proxy"
Oct 31 16:40:59.410017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.409507    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd98130-d605-4f94-8ba5-eb8afa6a8390" containerName="main"
Oct 31 16:40:59.414460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.414441    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452221    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452350    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452539 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452510    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452591    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452647    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452691    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452765    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.452831 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.452818    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkb4\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.510679 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.510649    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:40:59.553647 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553589    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkb4\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.553817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553675    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.553817 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553780    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.553962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553827    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.553962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553881    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.553962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553936    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.553991    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554064    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554092    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554330    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554396 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554370    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554634    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.554770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.554749    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.555064 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.555038    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.556966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.556938    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.557080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.557052    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.605986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.605952    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.606168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.605994    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkb4\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\") pod \"maas-default-gateway-openshift-default-fcb75c45f-mqgxj\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:40:59.727851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:40:59.727780    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:41:00.142730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:00.142698    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:41:00.181867 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:41:00.181832    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2d26ea_65dd_4a7c_a814_a15aefa8947f.slice/crio-92b6808d38c96d1dff5cbdd21a75e09ae8e5a24f634daaf3f8339974197b2d87 WatchSource:0}: Error finding container 92b6808d38c96d1dff5cbdd21a75e09ae8e5a24f634daaf3f8339974197b2d87: Status 404 returned error can't find the container with id 92b6808d38c96d1dff5cbdd21a75e09ae8e5a24f634daaf3f8339974197b2d87
Oct 31 16:41:00.184346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:00.184313    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:41:00.184480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:00.184383    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:41:00.184480 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:00.184411    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:41:01.172810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.172769    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" event={"ID":"0c2d26ea-65dd-4a7c-a814-a15aefa8947f","Type":"ContainerStarted","Data":"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"}
Oct 31 16:41:01.173242 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.172827    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" event={"ID":"0c2d26ea-65dd-4a7c-a814-a15aefa8947f","Type":"ContainerStarted","Data":"92b6808d38c96d1dff5cbdd21a75e09ae8e5a24f634daaf3f8339974197b2d87"}
Oct 31 16:41:01.328556 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.328507    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podStartSLOduration=2.328492964 podStartE2EDuration="2.328492964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:41:01.328097305 +0000 UTC m=+2306.389672491" watchObservedRunningTime="2026-10-31 16:41:01.328492964 +0000 UTC m=+2306.390068149"
Oct 31 16:41:01.728483 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.728445    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:41:01.730289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.730260    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:01.730420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:01.730311    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:02.728423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:02.728386    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:02.728868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:02.728443    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:02.973351 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:02.973312    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:41:03.732172 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:03.732122    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:03.732714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:03.732293    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:04.729001 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:04.728964    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:04.729187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:04.729016    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:05.728720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:05.728682    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:05.729110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:05.728735    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:06.728935 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:06.728891    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:06.729341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:06.728955    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:07.728414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:07.728371    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-mqgxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 31 16:41:07.728658 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:07.728448    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 31 16:41:08.733272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:08.733243    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:41:08.733676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:08.733647    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:41:08.734336 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:08.734315    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:41:12.312309 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.312277    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:41:12.312753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.312483    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" podUID="51bba813-c18f-4d27-b39f-b40fad3e81e8" containerName="maas-api" containerID="cri-o://118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307" gracePeriod=30
Oct 31 16:41:12.612289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.612193    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:41:12.767119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767085    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\" (UniqueName: \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\") pod \"51bba813-c18f-4d27-b39f-b40fad3e81e8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") "
Oct 31 16:41:12.767293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767279    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\") pod \"51bba813-c18f-4d27-b39f-b40fad3e81e8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") "
Oct 31 16:41:12.767366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767324    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\") pod \"51bba813-c18f-4d27-b39f-b40fad3e81e8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") "
Oct 31 16:41:12.767446 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767430    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") pod \"51bba813-c18f-4d27-b39f-b40fad3e81e8\" (UID: \"51bba813-c18f-4d27-b39f-b40fad3e81e8\") "
Oct 31 16:41:12.767951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767737    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca" pod "51bba813-c18f-4d27-b39f-b40fad3e81e8" (UID: "51bba813-c18f-4d27-b39f-b40fad3e81e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:12.767951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.767919    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51bba813-c18f-4d27-b39f-b40fad3e81e8-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:12.777039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.777009    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx" pod "51bba813-c18f-4d27-b39f-b40fad3e81e8" (UID: "51bba813-c18f-4d27-b39f-b40fad3e81e8"). InnerVolumeSpecName "kube-api-access-64dzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:12.777162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.777047    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs" pod "51bba813-c18f-4d27-b39f-b40fad3e81e8" (UID: "51bba813-c18f-4d27-b39f-b40fad3e81e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:12.778983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.778963    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls" pod "51bba813-c18f-4d27-b39f-b40fad3e81e8" (UID: "51bba813-c18f-4d27-b39f-b40fad3e81e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:12.869863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.869757    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:12.869863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.869797    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51bba813-c18f-4d27-b39f-b40fad3e81e8-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:12.869863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:12.869813    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dzx\" (UniqueName: \"kubernetes.io/projected/51bba813-c18f-4d27-b39f-b40fad3e81e8-kube-api-access-64dzx\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:13.250049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.249965    4417 generic.go:356] "Generic (PLEG): container finished" podID="51bba813-c18f-4d27-b39f-b40fad3e81e8" containerID="118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307" exitCode=0
Oct 31 16:41:13.250049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.250019    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"
Oct 31 16:41:13.250241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.250053    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" event={"ID":"51bba813-c18f-4d27-b39f-b40fad3e81e8","Type":"ContainerDied","Data":"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"}
Oct 31 16:41:13.250241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.250109    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8" event={"ID":"51bba813-c18f-4d27-b39f-b40fad3e81e8","Type":"ContainerDied","Data":"e29bb29194d91f80920662640a798879ee2b05de4f9d44a3506e8d553eedb0ae"}
Oct 31 16:41:13.250241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.250137    4417 scope.go:122] "RemoveContainer" containerID="118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"
Oct 31 16:41:13.259486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.259463    4417 scope.go:122] "RemoveContainer" containerID="118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"
Oct 31 16:41:13.259794 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:13.259774    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307\": container with ID starting with 118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307 not found: ID does not exist" containerID="118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"
Oct 31 16:41:13.259872 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.259816    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307"} err="failed to get container status \"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307\": rpc error: code = NotFound desc = could not find container \"118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307\": container with ID starting with 118084ea47e79212a2ea194cf170c591afaae4b6dad218f38843919d75590307 not found: ID does not exist"
Oct 31 16:41:13.406383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.406348    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:41:13.453152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:13.453114    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-4fe3a4-59cfcdf67-x62c8"]
Oct 31 16:41:15.126644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:15.125030    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bba813-c18f-4d27-b39f-b40fad3e81e8" path="/var/lib/kubelet/pods/51bba813-c18f-4d27-b39f-b40fad3e81e8/volumes"
Oct 31 16:41:16.339861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:16.339821    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:41:16.441528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:16.441483    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:41:21.340639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.340568    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" podUID="b623fcaa-64b7-41ab-93e7-a98727c4235e" containerName="payload-processing" containerID="cri-o://1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d" gracePeriod=25
Oct 31 16:41:21.442557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.442493    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" podUID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" containerName="payload-pre-processing" containerID="cri-o://def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b" gracePeriod=25
Oct 31 16:41:21.658892 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.658505    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:41:21.698306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.698275    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\") pod \"b623fcaa-64b7-41ab-93e7-a98727c4235e\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") "
Oct 31 16:41:21.698451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.698329    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\" (UniqueName: \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\") pod \"b623fcaa-64b7-41ab-93e7-a98727c4235e\" (UID: \"b623fcaa-64b7-41ab-93e7-a98727c4235e\") "
Oct 31 16:41:21.698736 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.698711    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume" pod "b623fcaa-64b7-41ab-93e7-a98727c4235e" (UID: "b623fcaa-64b7-41ab-93e7-a98727c4235e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:21.698887 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.698870    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b623fcaa-64b7-41ab-93e7-a98727c4235e-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:21.701119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.701088    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2" pod "b623fcaa-64b7-41ab-93e7-a98727c4235e" (UID: "b623fcaa-64b7-41ab-93e7-a98727c4235e"). InnerVolumeSpecName "kube-api-access-9prm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:21.758378 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.758351    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:41:21.799402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.799374    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\") pod \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") "
Oct 31 16:41:21.799589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.799411    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\" (UniqueName: \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\") pod \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\" (UID: \"feb5c016-a6dc-49fb-8974-d4c2fecba42f\") "
Oct 31 16:41:21.799791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.799772    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prm2\" (UniqueName: \"kubernetes.io/projected/b623fcaa-64b7-41ab-93e7-a98727c4235e-kube-api-access-9prm2\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:21.799896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.799807    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume" pod "feb5c016-a6dc-49fb-8974-d4c2fecba42f" (UID: "feb5c016-a6dc-49fb-8974-d4c2fecba42f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:21.801643 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.801584    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7" pod "feb5c016-a6dc-49fb-8974-d4c2fecba42f" (UID: "feb5c016-a6dc-49fb-8974-d4c2fecba42f"). InnerVolumeSpecName "kube-api-access-khhn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:21.900730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.900641    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb5c016-a6dc-49fb-8974-d4c2fecba42f-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:21.900730 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:21.900675    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhn7\" (UniqueName: \"kubernetes.io/projected/feb5c016-a6dc-49fb-8974-d4c2fecba42f-kube-api-access-khhn7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:22.347809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.347777    4417 generic.go:356] "Generic (PLEG): container finished" podID="b623fcaa-64b7-41ab-93e7-a98727c4235e" containerID="1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d" exitCode=0
Oct 31 16:41:22.348262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.347824    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"
Oct 31 16:41:22.348262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.347859    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" event={"ID":"b623fcaa-64b7-41ab-93e7-a98727c4235e","Type":"ContainerDied","Data":"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"}
Oct 31 16:41:22.348262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.347919    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx" event={"ID":"b623fcaa-64b7-41ab-93e7-a98727c4235e","Type":"ContainerDied","Data":"e0c3a92d5612225409bb56392101ed2301cd0b15acdf70296aa09a6d50882ffc"}
Oct 31 16:41:22.348262 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.347943    4417 scope.go:122] "RemoveContainer" containerID="1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"
Oct 31 16:41:22.349579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.349560    4417 generic.go:356] "Generic (PLEG): container finished" podID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" containerID="def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b" exitCode=0
Oct 31 16:41:22.349706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.349605    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" event={"ID":"feb5c016-a6dc-49fb-8974-d4c2fecba42f","Type":"ContainerDied","Data":"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"}
Oct 31 16:41:22.349706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.349675    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4" event={"ID":"feb5c016-a6dc-49fb-8974-d4c2fecba42f","Type":"ContainerDied","Data":"35b4b921953a60d2d17f75e06cb8f81a2aba24edbeeb62da3e7ba6b68553b257"}
Oct 31 16:41:22.349706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.349630    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"
Oct 31 16:41:22.366902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.366877    4417 scope.go:122] "RemoveContainer" containerID="1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"
Oct 31 16:41:22.367212 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:22.367192    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d\": container with ID starting with 1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d not found: ID does not exist" containerID="1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"
Oct 31 16:41:22.367271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.367219    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d"} err="failed to get container status \"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d\": rpc error: code = NotFound desc = could not find container \"1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d\": container with ID starting with 1aa334ac13075b6c99844dd8f2b049ad5f84b82d087989737cc98d60506e405d not found: ID does not exist"
Oct 31 16:41:22.367271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.367236    4417 scope.go:122] "RemoveContainer" containerID="def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"
Oct 31 16:41:22.375729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.375710    4417 scope.go:122] "RemoveContainer" containerID="def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"
Oct 31 16:41:22.375999 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:22.375979    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b\": container with ID starting with def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b not found: ID does not exist" containerID="def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"
Oct 31 16:41:22.376054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.376005    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b"} err="failed to get container status \"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b\": rpc error: code = NotFound desc = could not find container \"def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b\": container with ID starting with def366609c2a0bdc0d1d9d03d42e6330a6339e51393b6f37cc3a461565e8bb9b not found: ID does not exist"
Oct 31 16:41:22.500457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.500421    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:41:22.503231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.503205    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-f43174-56964f47-rqms4"]
Oct 31 16:41:22.645848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.645745    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:41:22.648392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:22.648363    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-f43174-565fcdf8cf-ws4zx"]
Oct 31 16:41:23.049193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:23.049160    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b623fcaa-64b7-41ab-93e7-a98727c4235e" path="/var/lib/kubelet/pods/b623fcaa-64b7-41ab-93e7-a98727c4235e/volumes"
Oct 31 16:41:23.049584 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:23.049571    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" path="/var/lib/kubelet/pods/feb5c016-a6dc-49fb-8974-d4c2fecba42f/volumes"
Oct 31 16:41:24.012115 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.012080    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:41:24.012555 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.012292    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerName="maas-api" containerID="cri-o://39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f" gracePeriod=30
Oct 31 16:41:24.141218 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.141184    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.47:8443/health\": dial tcp 10.133.0.47:8443: connect: connection refused"
Oct 31 16:41:24.332718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.332690    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:41:24.365443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.365411    4417 generic.go:356] "Generic (PLEG): container finished" podID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerID="39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f" exitCode=0
Oct 31 16:41:24.365659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.365456    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"
Oct 31 16:41:24.365659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.365509    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" event={"ID":"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f","Type":"ContainerDied","Data":"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"}
Oct 31 16:41:24.365659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.365559    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274" event={"ID":"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f","Type":"ContainerDied","Data":"3e012077801a60655b4cd744615bcd79abd0605c413195b6e9e0eb0f7f1fcd9e"}
Oct 31 16:41:24.365659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.365588    4417 scope.go:122] "RemoveContainer" containerID="39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"
Oct 31 16:41:24.376812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.376789    4417 scope.go:122] "RemoveContainer" containerID="39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"
Oct 31 16:41:24.377109 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:24.377085    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f\": container with ID starting with 39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f not found: ID does not exist" containerID="39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"
Oct 31 16:41:24.377220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.377115    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f"} err="failed to get container status \"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f\": rpc error: code = NotFound desc = could not find container \"39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f\": container with ID starting with 39b254cfcf462fb6dd9f3052b3c665e9082db2d1bd989708337d63c9b3c38b8f not found: ID does not exist"
Oct 31 16:41:24.440421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.438752    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\") pod \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") "
Oct 31 16:41:24.440421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.438807    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") pod \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") "
Oct 31 16:41:24.440421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.438961    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\") pod \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") "
Oct 31 16:41:24.440421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.438996    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\" (UniqueName: \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\") pod \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\" (UID: \"d5e05893-d2c1-48b6-ad63-1abc3eb72e3f\") "
Oct 31 16:41:24.448517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.446524    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca" pod "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" (UID: "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:24.451862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.451300    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs" pod "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" (UID: "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:24.451862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.451797    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf" pod "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" (UID: "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f"). InnerVolumeSpecName "kube-api-access-5h2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:24.452708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.452681    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls" pod "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" (UID: "d5e05893-d2c1-48b6-ad63-1abc3eb72e3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:24.540749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.540717    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:24.540749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.540743    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h2wf\" (UniqueName: \"kubernetes.io/projected/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-kube-api-access-5h2wf\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:24.540749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.540754    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:24.540982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.540764    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:24.778049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.778018    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:41:24.782377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:24.782351    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-59357a-85d48b48b7-q5274"]
Oct 31 16:41:25.054305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:25.054264    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" path="/var/lib/kubelet/pods/d5e05893-d2c1-48b6-ad63-1abc3eb72e3f/volumes"
Oct 31 16:41:29.600171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600137    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:41:29.600669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600654    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b623fcaa-64b7-41ab-93e7-a98727c4235e" containerName="payload-processing"
Oct 31 16:41:29.600723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600669    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b623fcaa-64b7-41ab-93e7-a98727c4235e" containerName="payload-processing"
Oct 31 16:41:29.600723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600685    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerName="maas-api"
Oct 31 16:41:29.600723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600691    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerName="maas-api"
Oct 31 16:41:29.600723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600701    4417 cpu_manager.go:414] "containerMap: removing container" podUID="51bba813-c18f-4d27-b39f-b40fad3e81e8" containerName="maas-api"
Oct 31 16:41:29.600723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600705    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bba813-c18f-4d27-b39f-b40fad3e81e8" containerName="maas-api"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600739    4417 cpu_manager.go:414] "containerMap: removing container" podUID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" containerName="payload-pre-processing"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600744    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" containerName="payload-pre-processing"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600823    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb5c016-a6dc-49fb-8974-d4c2fecba42f" containerName="payload-pre-processing"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600831    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b623fcaa-64b7-41ab-93e7-a98727c4235e" containerName="payload-processing"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600838    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e05893-d2c1-48b6-ad63-1abc3eb72e3f" containerName="maas-api"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600846    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bba813-c18f-4d27-b39f-b40fad3e81e8" containerName="maas-api"
Oct 31 16:41:29.600878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.600854    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3b69c1-72fb-45f5-81f3-4cceef64c2f8" containerName="istio-proxy"
Oct 31 16:41:29.605958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.605934    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.654441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.654396    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05db63\"/\"kube-root-ca.crt\""
Oct 31 16:41:29.700357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700330    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05db63\"/\"default-dockercfg-shldc\""
Oct 31 16:41:29.700529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700397    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05db63\"/\"openshift-service-ca.crt\""
Oct 31 16:41:29.700529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700421    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05db63\"/\"test-model-05db63-kserve-self-signed-certs\""
Oct 31 16:41:29.700661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700639    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.700742 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700727    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.700788 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700755    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.700830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700816    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.700875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700861    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.700987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.700966    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9t56\" (UniqueName: \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.703295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.703273    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:41:29.802202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802162    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802215    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802291    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802339    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802391 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802377    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802414    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t56\" (UniqueName: \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802600    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802756    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.802837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.802794    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.804671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.804650    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.804886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.804867    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.857316 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.857235    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t56\" (UniqueName: \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\") pod \"test-model-05db63-kserve-d85c46679-z8vzz\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") " pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:29.918453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:29.918413    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:30.285093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:30.285068    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:41:30.287810 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:41:30.287781    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73fdab20_9d9d_4a16_b5d0_277767873c2b.slice/crio-762af4b9db3ffb8411dd6181e37b73612db16db8165b054789c89cf23fbbb378 WatchSource:0}: Error finding container 762af4b9db3ffb8411dd6181e37b73612db16db8165b054789c89cf23fbbb378: Status 404 returned error can't find the container with id 762af4b9db3ffb8411dd6181e37b73612db16db8165b054789c89cf23fbbb378
Oct 31 16:41:30.401420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:30.401389    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" event={"ID":"73fdab20-9d9d-4a16-b5d0-277767873c2b","Type":"ContainerStarted","Data":"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"}
Oct 31 16:41:30.401588 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:30.401437    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" event={"ID":"73fdab20-9d9d-4a16-b5d0-277767873c2b","Type":"ContainerStarted","Data":"762af4b9db3ffb8411dd6181e37b73612db16db8165b054789c89cf23fbbb378"}
Oct 31 16:41:30.506891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:30.506839    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" podStartSLOduration=1.5068231650000001 podStartE2EDuration="1.506823165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:41:30.505857286 +0000 UTC m=+2335.567432471" watchObservedRunningTime="2026-10-31 16:41:30.506823165 +0000 UTC m=+2335.568398344"
Oct 31 16:41:30.919420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:30.919388    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:32.945313 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:41:32.945270    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:41:37.349118 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.349081    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:41:37.357047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.357020    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.451265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.451232    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-66db1c\"/\"openshift-service-ca.crt\""
Oct 31 16:41:37.451455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.451274    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-66db1c\"/\"test-model-66db1c-kserve-self-signed-certs\""
Oct 31 16:41:37.451455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.451280    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-66db1c\"/\"kube-root-ca.crt\""
Oct 31 16:41:37.451455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.451340    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-66db1c\"/\"default-dockercfg-957df\""
Oct 31 16:41:37.453848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.453824    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:41:37.513983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.513942    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrxsj\" (UniqueName: \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.519046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.519010    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.519313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.519283    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.519524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.519509    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.519739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.519724    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.519914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.519891    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621360    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621425    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621451    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621512    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrxsj\" (UniqueName: \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621545    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621711 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621573    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.621954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621929    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.622019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.621982    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.622019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.622009    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.623710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.623685    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.623983 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.623964    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.673838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.673805    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrxsj\" (UniqueName: \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\") pod \"test-model-66db1c-kserve-7cf9f787b7-c4gwh\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") " pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:37.969379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:37.969297    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:38.335737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:38.335708    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:41:38.338841 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:41:38.338813    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aa7f72a_50a1_43ee_a0a3_e02797b09c8e.slice/crio-3cc581f10c51e917ce073b601288f77625c28c7f5fbe0b269b8cf7a37c7b07fd WatchSource:0}: Error finding container 3cc581f10c51e917ce073b601288f77625c28c7f5fbe0b269b8cf7a37c7b07fd: Status 404 returned error can't find the container with id 3cc581f10c51e917ce073b601288f77625c28c7f5fbe0b269b8cf7a37c7b07fd
Oct 31 16:41:38.448657 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:38.448602    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" event={"ID":"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e","Type":"ContainerStarted","Data":"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"}
Oct 31 16:41:38.449057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:38.448665    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" event={"ID":"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e","Type":"ContainerStarted","Data":"3cc581f10c51e917ce073b601288f77625c28c7f5fbe0b269b8cf7a37c7b07fd"}
Oct 31 16:41:38.555738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:38.555685    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" podStartSLOduration=1.555667765 podStartE2EDuration="1.555667765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:41:38.554727062 +0000 UTC m=+2343.616302248" watchObservedRunningTime="2026-10-31 16:41:38.555667765 +0000 UTC m=+2343.617242949"
Oct 31 16:41:38.969681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:38.969647    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:39.919546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:39.919514    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:39.932322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:39.932297    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:40.471849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:40.471825    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:41:45.187876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.187840    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:41:45.188284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.188044    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" podUID="c494b161-faae-4aba-ad28-1606655a9aae" containerName="maas-api" containerID="cri-o://bd92d8125acf7bf10369ac2a301aa9646225a206e6ed237405ec1330846a9cec" gracePeriod=30
Oct 31 16:41:45.489231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.489205    4417 generic.go:356] "Generic (PLEG): container finished" podID="c494b161-faae-4aba-ad28-1606655a9aae" containerID="bd92d8125acf7bf10369ac2a301aa9646225a206e6ed237405ec1330846a9cec" exitCode=0
Oct 31 16:41:45.489397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.489288    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" event={"ID":"c494b161-faae-4aba-ad28-1606655a9aae","Type":"ContainerDied","Data":"bd92d8125acf7bf10369ac2a301aa9646225a206e6ed237405ec1330846a9cec"}
Oct 31 16:41:45.489397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.489359    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7" event={"ID":"c494b161-faae-4aba-ad28-1606655a9aae","Type":"ContainerDied","Data":"1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9"}
Oct 31 16:41:45.489397 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.489383    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1353a6030ed54faa05f45d372de1747ab21487280cf8cbc90d6fb05f9cd031d9"
Oct 31 16:41:45.492953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.492931    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:41:45.625057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.625022    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") pod \"c494b161-faae-4aba-ad28-1606655a9aae\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") "
Oct 31 16:41:45.625542 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.625517    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\" (UniqueName: \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\") pod \"c494b161-faae-4aba-ad28-1606655a9aae\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") "
Oct 31 16:41:45.625811 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.625770    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\") pod \"c494b161-faae-4aba-ad28-1606655a9aae\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") "
Oct 31 16:41:45.626061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.626040    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\") pod \"c494b161-faae-4aba-ad28-1606655a9aae\" (UID: \"c494b161-faae-4aba-ad28-1606655a9aae\") "
Oct 31 16:41:45.627319 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.627280    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca" pod "c494b161-faae-4aba-ad28-1606655a9aae" (UID: "c494b161-faae-4aba-ad28-1606655a9aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:41:45.631590 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.631554    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv" pod "c494b161-faae-4aba-ad28-1606655a9aae" (UID: "c494b161-faae-4aba-ad28-1606655a9aae"). InnerVolumeSpecName "kube-api-access-rdpvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:41:45.634668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.634438    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs" pod "c494b161-faae-4aba-ad28-1606655a9aae" (UID: "c494b161-faae-4aba-ad28-1606655a9aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:45.636806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.636772    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls" pod "c494b161-faae-4aba-ad28-1606655a9aae" (UID: "c494b161-faae-4aba-ad28-1606655a9aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:41:45.735259 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.731689    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:45.735533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.735503    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpvv\" (UniqueName: \"kubernetes.io/projected/c494b161-faae-4aba-ad28-1606655a9aae-kube-api-access-rdpvv\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:45.735722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.735697    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c494b161-faae-4aba-ad28-1606655a9aae-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:45.736449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:45.736423    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c494b161-faae-4aba-ad28-1606655a9aae-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:41:46.494498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:46.494466    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"
Oct 31 16:41:46.647513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:46.647479    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:41:46.693371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:46.693336    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-f43174-6554664877-ltrd7"]
Oct 31 16:41:47.049321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:47.049285    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c494b161-faae-4aba-ad28-1606655a9aae" path="/var/lib/kubelet/pods/c494b161-faae-4aba-ad28-1606655a9aae/volumes"
Oct 31 16:41:47.450933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:47.450851    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:41:47.969496 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:47.969456    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:47.981911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:47.981880    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:48.516268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:48.516239    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:41:59.571725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:41:59.571683    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:42:02.475885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.475838    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main" containerID="cri-o://985be2299149b2afadbaec9fa35749ca8ddf1edccf20bbc1d9410f40b982ac29" gracePeriod=45
Oct 31 16:42:02.617561 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.617531    4417 generic.go:356] "Generic (PLEG): container finished" podID="a908cd75-83cc-403d-998b-188faa247149" containerID="985be2299149b2afadbaec9fa35749ca8ddf1edccf20bbc1d9410f40b982ac29" exitCode=0
Oct 31 16:42:02.617792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.617641    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" event={"ID":"a908cd75-83cc-403d-998b-188faa247149","Type":"ContainerDied","Data":"985be2299149b2afadbaec9fa35749ca8ddf1edccf20bbc1d9410f40b982ac29"}
Oct 31 16:42:02.774029 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.774003    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:42:02.847414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847371    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\" (UniqueName: \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.847607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847457    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\" (UniqueName: \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.847607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847545    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.847607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847592    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.847829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847772    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.847829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847815    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\") pod \"a908cd75-83cc-403d-998b-188faa247149\" (UID: \"a908cd75-83cc-403d-998b-188faa247149\") "
Oct 31 16:42:02.848018 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.847991    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.848232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.848208    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.848303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.848232    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.848460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.848441    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.848541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.848465    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.848541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.848481    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.859703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.853778    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "kube-api-access-pr5zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:02.859703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.853858    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:02.859703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.854941    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm" pod "a908cd75-83cc-403d-998b-188faa247149" (UID: "a908cd75-83cc-403d-998b-188faa247149"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:02.955651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.949452    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a908cd75-83cc-403d-998b-188faa247149-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.955651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.949628    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a908cd75-83cc-403d-998b-188faa247149-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.955651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:02.949647    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5zs\" (UniqueName: \"kubernetes.io/projected/a908cd75-83cc-403d-998b-188faa247149-kube-api-access-pr5zs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:02.955651 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:02.949653    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:42:03.624447 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:03.624416    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"
Oct 31 16:42:03.624933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:03.624418    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88" event={"ID":"a908cd75-83cc-403d-998b-188faa247149","Type":"ContainerDied","Data":"582b2e3f6c5e44157cf423199073cde99a627f0ee6b9ce1adf10f172b9c705c6"}
Oct 31 16:42:03.624933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:03.624536    4417 scope.go:122] "RemoveContainer" containerID="985be2299149b2afadbaec9fa35749ca8ddf1edccf20bbc1d9410f40b982ac29"
Oct 31 16:42:03.774838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:03.774800    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:42:03.778360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:03.778331    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-2-w0-ac1273-kserve-589b7f7c59-m7p88"]
Oct 31 16:42:05.052770 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:05.052734    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a908cd75-83cc-403d-998b-188faa247149" path="/var/lib/kubelet/pods/a908cd75-83cc-403d-998b-188faa247149/volumes"
Oct 31 16:42:06.693814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:06.693776    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:42:11.694367 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:11.694326    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing" containerID="cri-o://3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53" gracePeriod=25
Oct 31 16:42:14.596768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:14.596710    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main" containerID="cri-o://507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0" gracePeriod=45
Oct 31 16:42:14.860784 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:14.860715    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:42:15.002653 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.002601    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.002834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.002705    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.002834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.002794    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.002949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.002905    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.003011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.002968    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.003073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.003027    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\" (UniqueName: \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\") pod \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\" (UID: \"941784d0-03b0-4dd7-9c5f-5d5ba03827e8\") "
Oct 31 16:42:15.003975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.003944    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.004167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.004145    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.004342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.004322    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.007235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.007208    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:15.014808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.014778    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"]
Oct 31 16:42:15.015595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015573    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main"
Oct 31 16:42:15.015668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015599    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main"
Oct 31 16:42:15.015668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015643    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c494b161-faae-4aba-ad28-1606655a9aae" containerName="maas-api"
Oct 31 16:42:15.015668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015652    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c494b161-faae-4aba-ad28-1606655a9aae" containerName="maas-api"
Oct 31 16:42:15.015668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015666    4417 cpu_manager.go:414] "containerMap: removing container" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main"
Oct 31 16:42:15.015805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015675    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main"
Oct 31 16:42:15.015859 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015845    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a908cd75-83cc-403d-998b-188faa247149" containerName="main"
Oct 31 16:42:15.015893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015865    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c494b161-faae-4aba-ad28-1606655a9aae" containerName="maas-api"
Oct 31 16:42:15.015893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.015874    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerName="main"
Oct 31 16:42:15.017981 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.017954    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:15.018793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.018763    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz" pod "941784d0-03b0-4dd7-9c5f-5d5ba03827e8" (UID: "941784d0-03b0-4dd7-9c5f-5d5ba03827e8"). InnerVolumeSpecName "kube-api-access-fxjrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:15.020114 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.020094    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.068580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.068551    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:42:15.068794 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.068771    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:42:15.080419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.080395    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"]
Oct 31 16:42:15.104060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104032    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.104181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104099    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfs5\" (UniqueName: \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.104235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104184    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.104372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104357    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.104442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104381    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.104442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104397    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.104442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104412    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.104442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104425    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.104442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.104438    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxjrz\" (UniqueName: \"kubernetes.io/projected/941784d0-03b0-4dd7-9c5f-5d5ba03827e8-kube-api-access-fxjrz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:15.206273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.206177    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.206435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.206276    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfs5\" (UniqueName: \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.206435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.206334    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.207272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.207235    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.209422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.209387    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.258395 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.258366    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfs5\" (UniqueName: \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.359195 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.359160    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:15.693270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.693240    4417 generic.go:356] "Generic (PLEG): container finished" podID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" containerID="507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0" exitCode=0
Oct 31 16:42:15.693741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.693287    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"
Oct 31 16:42:15.693741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.693300    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" event={"ID":"941784d0-03b0-4dd7-9c5f-5d5ba03827e8","Type":"ContainerDied","Data":"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"}
Oct 31 16:42:15.693741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.693331    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8" event={"ID":"941784d0-03b0-4dd7-9c5f-5d5ba03827e8","Type":"ContainerDied","Data":"7b0697dce5cffd99564a75869d8054797e113ac8ebdccc2074373d5773406da8"}
Oct 31 16:42:15.693741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.693346    4417 scope.go:122] "RemoveContainer" containerID="507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"
Oct 31 16:42:15.702358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.702335    4417 scope.go:122] "RemoveContainer" containerID="507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"
Oct 31 16:42:15.702601 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:15.702581    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0\": container with ID starting with 507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0 not found: ID does not exist" containerID="507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"
Oct 31 16:42:15.702686 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.702609    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0"} err="failed to get container status \"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0\": rpc error: code = NotFound desc = could not find container \"507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0\": container with ID starting with 507a90d5987f4ac5affa2eed4c3dd48e673b127117bb1b502b4e38daf10fbaf0 not found: ID does not exist"
Oct 31 16:42:15.729211 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.729183    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"]
Oct 31 16:42:15.734556 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:42:15.734527    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee0b0a3_9ed2_423b_aa4a_b1eb7a287cfb.slice/crio-43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd WatchSource:0}: Error finding container 43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd: Status 404 returned error can't find the container with id 43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd
Oct 31 16:42:15.869166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.869134    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:42:15.886876 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:15.886757    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/distinct-2-w5-ef6d93-kserve-569677974b-brwd8"]
Oct 31 16:42:16.701059 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:16.701030    4417 generic.go:356] "Generic (PLEG): container finished" podID="0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" containerID="e55145a795da0d6843dd11fb540bcadff701d212472a1864150a3076a7de522f" exitCode=0
Oct 31 16:42:16.701476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:16.701115    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98" event={"ID":"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb","Type":"ContainerDied","Data":"e55145a795da0d6843dd11fb540bcadff701d212472a1864150a3076a7de522f"}
Oct 31 16:42:16.701476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:16.701162    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98" event={"ID":"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb","Type":"ContainerStarted","Data":"43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd"}
Oct 31 16:42:17.051704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.051673    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941784d0-03b0-4dd7-9c5f-5d5ba03827e8" path="/var/lib/kubelet/pods/941784d0-03b0-4dd7-9c5f-5d5ba03827e8/volumes"
Oct 31 16:42:17.843729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.843704    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:17.940809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.940778    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\") pod \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") "
Oct 31 16:42:17.940965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.940828    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\") pod \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") "
Oct 31 16:42:17.941077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.941061    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\" (UniqueName: \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\") pod \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\" (UID: \"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb\") "
Oct 31 16:42:17.941245 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.941218    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca" pod "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" (UID: "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:17.941504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.941483    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:17.943102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.943077    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca" pod "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" (UID: "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:42:17.943223 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:17.943097    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5" pod "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" (UID: "0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb"). InnerVolumeSpecName "kube-api-access-jmfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:18.042989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.042961    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfs5\" (UniqueName: \"kubernetes.io/projected/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-kube-api-access-jmfs5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.042989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.042989    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:18.717414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.717383    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"
Oct 31 16:42:18.717586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.717411    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98" event={"ID":"0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb","Type":"ContainerDied","Data":"43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd"}
Oct 31 16:42:18.717586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.717456    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43603b11b5ddaed6dddc73bd9652b822bf91847f85b5a6c6dda07f05417cfbdd"
Oct 31 16:42:18.779176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:18.779119    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:26.452914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:26.452879    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:42:26.551459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:26.551422    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:42:28.778011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:28.777964    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:31.454031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.453993    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing" containerID="cri-o://aaa887874db9c94ba4f8d31a2fc9659d9c4d234b6ee86f2eda3fa1f9b6521bb2" gracePeriod=25
Oct 31 16:42:31.552804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.552731    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing" containerID="cri-o://78e7e270a9fb0fb4af1831d37602d6b662ffec1703ce565758c8b487edbc3ccd" gracePeriod=25
Oct 31 16:42:31.643660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.643608    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:31.643808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.643687    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:31.954127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.954095    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:42:31.987628 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.987557    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\") pod \"4727544f-ec71-4aa1-96cd-289815522f8e\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") "
Oct 31 16:42:31.987757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.987738    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\" (UniqueName: \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\") pod \"4727544f-ec71-4aa1-96cd-289815522f8e\" (UID: \"4727544f-ec71-4aa1-96cd-289815522f8e\") "
Oct 31 16:42:31.987912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.987890    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume" pod "4727544f-ec71-4aa1-96cd-289815522f8e" (UID: "4727544f-ec71-4aa1-96cd-289815522f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:31.988022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.988008    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4727544f-ec71-4aa1-96cd-289815522f8e-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:31.989777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:31.989755    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v" pod "4727544f-ec71-4aa1-96cd-289815522f8e" (UID: "4727544f-ec71-4aa1-96cd-289815522f8e"). InnerVolumeSpecName "kube-api-access-8wx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:32.089537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.089498    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wx2v\" (UniqueName: \"kubernetes.io/projected/4727544f-ec71-4aa1-96cd-289815522f8e-kube-api-access-8wx2v\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:32.797540 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.797509    4417 generic.go:356] "Generic (PLEG): container finished" podID="4727544f-ec71-4aa1-96cd-289815522f8e" containerID="3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53" exitCode=0
Oct 31 16:42:32.797997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.797586    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"
Oct 31 16:42:32.797997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.797599    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" event={"ID":"4727544f-ec71-4aa1-96cd-289815522f8e","Type":"ContainerDied","Data":"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"}
Oct 31 16:42:32.797997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.797683    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95" event={"ID":"4727544f-ec71-4aa1-96cd-289815522f8e","Type":"ContainerDied","Data":"834b8e4c0cf0ce03a3c53a174effe460180f1e3f15cbaa4fdd6daea329645295"}
Oct 31 16:42:32.797997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.797707    4417 scope.go:122] "RemoveContainer" containerID="3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"
Oct 31 16:42:32.806791 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.806773    4417 scope.go:122] "RemoveContainer" containerID="3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"
Oct 31 16:42:32.807062 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:32.807044    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53\": container with ID starting with 3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53 not found: ID does not exist" containerID="3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"
Oct 31 16:42:32.807113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.807070    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53"} err="failed to get container status \"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53\": rpc error: code = NotFound desc = could not find container \"3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53\": container with ID starting with 3e432f25c6e8dd582793c4fa0fd42557d72dc76675aeaa2b07644c4a7ae46e53 not found: ID does not exist"
Oct 31 16:42:32.944231 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:32.944197    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:42:32.953600 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.953568    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:42:32.994193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:32.994166    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6d93-6b75448499-hll95"]
Oct 31 16:42:33.051273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:33.051194    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" path="/var/lib/kubelet/pods/4727544f-ec71-4aa1-96cd-289815522f8e/volumes"
Oct 31 16:42:36.370436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:36.370406    4417 scope.go:122] "RemoveContainer" containerID="bd92d8125acf7bf10369ac2a301aa9646225a206e6ed237405ec1330846a9cec"
Oct 31 16:42:41.454703 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:41.454671    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:42:41.455280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:41.454943    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy" containerID="cri-o://f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff" gracePeriod=30
Oct 31 16:42:41.643906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:41.643864    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:41.644078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:41.643907    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:42:46.717477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.717450    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:42:46.858948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.858917    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.858985    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859025    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859051    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859114    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859266    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859311    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859345    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859370    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:46.859526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859376    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\") pod \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\" (UID: \"6218c303-4ea9-45b7-ad43-17c0b7ee2bff\") "
Oct 31 16:42:46.859689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859667    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:46.859934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.859908    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:46.860148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.860071    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:46.860148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.860065    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.860334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.860166    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.860437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.860399    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:46.861810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.861778    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:46.862058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.862032    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:42:46.862153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.862138    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "kube-api-access-b8dcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:46.862493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.862477    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo" pod "6218c303-4ea9-45b7-ad43-17c0b7ee2bff" (UID: "6218c303-4ea9-45b7-ad43-17c0b7ee2bff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:42:46.876601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.876560    4417 generic.go:356] "Generic (PLEG): container finished" podID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerID="f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff" exitCode=0
Oct 31 16:42:46.876727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.876636    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"
Oct 31 16:42:46.876727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.876653    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerDied","Data":"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"}
Oct 31 16:42:46.876727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.876723    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx" event={"ID":"6218c303-4ea9-45b7-ad43-17c0b7ee2bff","Type":"ContainerDied","Data":"d74dd52e151653cad93d266945a1ba32aea9bb630d216fc4684dfe43fdae5e65"}
Oct 31 16:42:46.876882 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.876755    4417 scope.go:122] "RemoveContainer" containerID="f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"
Oct 31 16:42:46.892424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.892406    4417 scope.go:122] "RemoveContainer" containerID="32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"
Oct 31 16:42:46.900698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.900682    4417 scope.go:122] "RemoveContainer" containerID="f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"
Oct 31 16:42:46.900963 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:46.900945    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff\": container with ID starting with f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff not found: ID does not exist" containerID="f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"
Oct 31 16:42:46.901027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.900970    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff"} err="failed to get container status \"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff\": rpc error: code = NotFound desc = could not find container \"f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff\": container with ID starting with f982030bf8c3492082db4327a9c24898b134e88bfebe90908631c2e2578626ff not found: ID does not exist"
Oct 31 16:42:46.901027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.900988    4417 scope.go:122] "RemoveContainer" containerID="32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"
Oct 31 16:42:46.901230 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:42:46.901210    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb\": container with ID starting with 32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb not found: ID does not exist" containerID="32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"
Oct 31 16:42:46.901272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.901236    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb"} err="failed to get container status \"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb\": rpc error: code = NotFound desc = could not find container \"32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb\": container with ID starting with 32a3c88acf089f4584cee43589b077c76768176645e192d010222bb3e133c5eb not found: ID does not exist"
Oct 31 16:42:46.961120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961087    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961116    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961128    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dcz\" (UniqueName: \"kubernetes.io/projected/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-kube-api-access-b8dcz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961138    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961148    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961156    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:46.961443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:46.961166    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6218c303-4ea9-45b7-ad43-17c0b7ee2bff-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:47.028060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:47.028028    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:42:47.033093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:47.033051    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-x5xsx"]
Oct 31 16:42:47.049289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:47.049256    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" path="/var/lib/kubelet/pods/6218c303-4ea9-45b7-ad43-17c0b7ee2bff/volumes"
Oct 31 16:42:51.665641 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:42:51.644172    4417 logging.go:55] [core] [Channel #6767 SubChannel #6769]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: connect: connection refused"
Oct 31 16:42:51.665641 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:42:51.644217    4417 logging.go:55] [core] [Channel #6768 SubChannel #6770]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.39:50052", ServerName: "10.133.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.39:50052: connect: connection refused"
Oct 31 16:42:51.930773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:51.930499    4417 generic.go:356] "Generic (PLEG): container finished" podID="faa484d9-13c0-4a05-b493-786a96446e90" containerID="78e7e270a9fb0fb4af1831d37602d6b662ffec1703ce565758c8b487edbc3ccd" exitCode=0
Oct 31 16:42:51.930773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:51.930690    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" event={"ID":"faa484d9-13c0-4a05-b493-786a96446e90","Type":"ContainerDied","Data":"78e7e270a9fb0fb4af1831d37602d6b662ffec1703ce565758c8b487edbc3ccd"}
Oct 31 16:42:51.932980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:51.932955    4417 generic.go:356] "Generic (PLEG): container finished" podID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerID="aaa887874db9c94ba4f8d31a2fc9659d9c4d234b6ee86f2eda3fa1f9b6521bb2" exitCode=0
Oct 31 16:42:51.933203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:51.933166    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" event={"ID":"3d4c0100-b01f-4d6f-a68d-40a459879e34","Type":"ContainerDied","Data":"aaa887874db9c94ba4f8d31a2fc9659d9c4d234b6ee86f2eda3fa1f9b6521bb2"}
Oct 31 16:42:52.363081 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.359365    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:42:52.426651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.426603    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:42:52.434723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.434694    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\" (UniqueName: \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\") pod \"faa484d9-13c0-4a05-b493-786a96446e90\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") "
Oct 31 16:42:52.434857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.434753    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\") pod \"faa484d9-13c0-4a05-b493-786a96446e90\" (UID: \"faa484d9-13c0-4a05-b493-786a96446e90\") "
Oct 31 16:42:52.434857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.434810    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\" (UniqueName: \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\") pod \"3d4c0100-b01f-4d6f-a68d-40a459879e34\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") "
Oct 31 16:42:52.434977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.434881    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\") pod \"3d4c0100-b01f-4d6f-a68d-40a459879e34\" (UID: \"3d4c0100-b01f-4d6f-a68d-40a459879e34\") "
Oct 31 16:42:52.435346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.435303    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume" pod "3d4c0100-b01f-4d6f-a68d-40a459879e34" (UID: "3d4c0100-b01f-4d6f-a68d-40a459879e34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:52.435607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.435581    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume" pod "faa484d9-13c0-4a05-b493-786a96446e90" (UID: "faa484d9-13c0-4a05-b493-786a96446e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:42:52.436830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.436802    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc" pod "faa484d9-13c0-4a05-b493-786a96446e90" (UID: "faa484d9-13c0-4a05-b493-786a96446e90"). InnerVolumeSpecName "kube-api-access-vcdpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:52.437668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.437647    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq" pod "3d4c0100-b01f-4d6f-a68d-40a459879e34" (UID: "3d4c0100-b01f-4d6f-a68d-40a459879e34"). InnerVolumeSpecName "kube-api-access-l5nnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:42:52.536301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.536267    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4c0100-b01f-4d6f-a68d-40a459879e34-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:52.536301 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.536304    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdpc\" (UniqueName: \"kubernetes.io/projected/faa484d9-13c0-4a05-b493-786a96446e90-kube-api-access-vcdpc\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:52.536494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.536321    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa484d9-13c0-4a05-b493-786a96446e90-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:52.536494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.536339    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nnq\" (UniqueName: \"kubernetes.io/projected/3d4c0100-b01f-4d6f-a68d-40a459879e34-kube-api-access-l5nnq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:42:52.644879 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.644770    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:50052\" within 1s: context deadline exceeded"
Oct 31 16:42:52.645052 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:42:52.644878    4417 logging.go:55] [core] [Channel #6767 SubChannel #6769]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: operation was canceled"
Oct 31 16:42:52.645052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.644887    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:42:52.645052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.644886    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.39:50052\" within 1s: context deadline exceeded"
Oct 31 16:42:52.645052 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:42:52.644938    4417 logging.go:55] [core] [Channel #6768 SubChannel #6770]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.39:50052", ServerName: "10.133.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.39:50052: operation was canceled"
Oct 31 16:42:52.645052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.644991    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:42:52.944482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.944392    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"
Oct 31 16:42:52.944482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.944410    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv" event={"ID":"faa484d9-13c0-4a05-b493-786a96446e90","Type":"ContainerDied","Data":"baa676f8a066a5f10d60f88207c6d0e41c8777dc57b59b2148569f7f9fab8f11"}
Oct 31 16:42:52.945047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.944481    4417 scope.go:122] "RemoveContainer" containerID="78e7e270a9fb0fb4af1831d37602d6b662ffec1703ce565758c8b487edbc3ccd"
Oct 31 16:42:52.946444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.946423    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7" event={"ID":"3d4c0100-b01f-4d6f-a68d-40a459879e34","Type":"ContainerDied","Data":"ae8998c53920b7d2a4115f9c2d1b0d7319a251ca59b98af44b65b9df19157b27"}
Oct 31 16:42:52.946589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.946516    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"
Oct 31 16:42:52.957109 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:52.957088    4417 scope.go:122] "RemoveContainer" containerID="aaa887874db9c94ba4f8d31a2fc9659d9c4d234b6ee86f2eda3fa1f9b6521bb2"
Oct 31 16:42:53.095428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:53.095396    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:42:53.141498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:53.141401    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a1685e-cfbd66747-h6jcv"]
Oct 31 16:42:53.200761 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:53.200686    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:42:53.207306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:53.207278    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a1685e-7f8694fc87-ssvs7"]
Oct 31 16:42:55.052375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:55.052340    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" path="/var/lib/kubelet/pods/3d4c0100-b01f-4d6f-a68d-40a459879e34/volumes"
Oct 31 16:42:55.052960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:55.052940    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa484d9-13c0-4a05-b493-786a96446e90" path="/var/lib/kubelet/pods/faa484d9-13c0-4a05-b493-786a96446e90/volumes"
Oct 31 16:42:57.815264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:57.815232    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"]
Oct 31 16:42:57.861144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:57.861099    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1685e-g8t98"]
Oct 31 16:42:59.048251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:42:59.048217    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" path="/var/lib/kubelet/pods/0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb/volumes"
Oct 31 16:43:02.945181 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:43:02.945138    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:43:14.273191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:14.273160    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:43:14.273766 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:14.273353    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy" containerID="cri-o://1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146" gracePeriod=30
Oct 31 16:43:19.550100 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.550041    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:43:19.741286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741257    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741292    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741321    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741352    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741416    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741546 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741484    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741549    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.741769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741549    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:19.741769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741593    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:19.741769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741754    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.742011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741803    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\") pod \"f01dac0c-4523-4f5a-b18a-653d540abae0\" (UID: \"f01dac0c-4523-4f5a-b18a-653d540abae0\") "
Oct 31 16:43:19.742011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741871    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:19.742011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.741908    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:43:19.742196 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742089    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:19.742299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742279    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.742371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742303    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.742371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742318    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.742371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742333    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.742371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.742349    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01dac0c-4523-4f5a-b18a-653d540abae0-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.743767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.743740    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:43:19.743971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.743889    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:43:19.743971 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.743951    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "kube-api-access-pwjhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:43:19.744179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.744163    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token" pod "f01dac0c-4523-4f5a-b18a-653d540abae0" (UID: "f01dac0c-4523-4f5a-b18a-653d540abae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:43:19.843659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.843559    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.843659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.843589    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.843659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.843600    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01dac0c-4523-4f5a-b18a-653d540abae0-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:19.843659 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:19.843610    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjhf\" (UniqueName: \"kubernetes.io/projected/f01dac0c-4523-4f5a-b18a-653d540abae0-kube-api-access-pwjhf\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:43:20.098593 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.098497    4417 generic.go:356] "Generic (PLEG): container finished" podID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerID="1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146" exitCode=0
Oct 31 16:43:20.098813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.098695    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" event={"ID":"f01dac0c-4523-4f5a-b18a-653d540abae0","Type":"ContainerDied","Data":"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"}
Oct 31 16:43:20.098813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.098786    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk" event={"ID":"f01dac0c-4523-4f5a-b18a-653d540abae0","Type":"ContainerDied","Data":"3e8b4984da62ae9bc0f8314269598b0364fc7ef317c98a34700bf6ed0b0ecf27"}
Oct 31 16:43:20.098956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.098828    4417 scope.go:122] "RemoveContainer" containerID="1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"
Oct 31 16:43:20.099280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.099255    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"
Oct 31 16:43:20.111372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.111344    4417 scope.go:122] "RemoveContainer" containerID="1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"
Oct 31 16:43:20.111692 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:43:20.111610    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146\": container with ID starting with 1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146 not found: ID does not exist" containerID="1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"
Oct 31 16:43:20.111692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.111661    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146"} err="failed to get container status \"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146\": rpc error: code = NotFound desc = could not find container \"1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146\": container with ID starting with 1c0c95b4db2ceddefb015109c9f1e276b7e570568bef5bdac5410116f1381146 not found: ID does not exist"
Oct 31 16:43:20.251271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.251240    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:43:20.297413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:20.297382    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-cqlbk"]
Oct 31 16:43:21.050533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:43:21.050497    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" path="/var/lib/kubelet/pods/f01dac0c-4523-4f5a-b18a-653d540abae0/volumes"
Oct 31 16:43:32.944550 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:43:32.944506    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:44:01.570418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.570332    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:44:01.571030 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.570547    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerName="discovery" containerID="cri-o://fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6" gracePeriod=30
Oct 31 16:44:01.817902 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.817877    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:44:01.883645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883536    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883576    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883645 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883637    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883706    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883755    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883846    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.883970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.883874    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\") pod \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\" (UID: \"84a0df2e-35fe-4c53-90a8-fb3e18d60366\") "
Oct 31 16:44:01.884224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.884191    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:44:01.884374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.884353    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-ca-configmap\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.885997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.885973    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:44:01.885997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.885986    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:44:01.886291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.886270    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:44:01.886380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.886289    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "kube-api-access-r8xm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:44:01.886380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.886367    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:44:01.886463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.886430    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts" pod "84a0df2e-35fe-4c53-90a8-fb3e18d60366" (UID: "84a0df2e-35fe-4c53-90a8-fb3e18d60366"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:44:01.985051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985015    4417 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-csr-dns-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.985051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985043    4417 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-cacerts\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.985051 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985055    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8xm4\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-kube-api-access-r8xm4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.985357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985069    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.985357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985079    4417 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a0df2e-35fe-4c53-90a8-fb3e18d60366-local-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:01.985357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:01.985087    4417 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a0df2e-35fe-4c53-90a8-fb3e18d60366-istio-kubeconfig\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:44:02.334717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.334683    4417 generic.go:356] "Generic (PLEG): container finished" podID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerID="fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6" exitCode=0
Oct 31 16:44:02.334907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.334742    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"
Oct 31 16:44:02.334907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.334769    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" event={"ID":"84a0df2e-35fe-4c53-90a8-fb3e18d60366","Type":"ContainerDied","Data":"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"}
Oct 31 16:44:02.334907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.334821    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp" event={"ID":"84a0df2e-35fe-4c53-90a8-fb3e18d60366","Type":"ContainerDied","Data":"55d4e788e8b13a99e371612b582b5ca0fc58058725cab03bef800c8f2896ac2a"}
Oct 31 16:44:02.334907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.334841    4417 scope.go:122] "RemoveContainer" containerID="fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"
Oct 31 16:44:02.349058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.349038    4417 scope.go:122] "RemoveContainer" containerID="fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"
Oct 31 16:44:02.349334 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:44:02.349315    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6\": container with ID starting with fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6 not found: ID does not exist" containerID="fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"
Oct 31 16:44:02.349406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.349344    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6"} err="failed to get container status \"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6\": rpc error: code = NotFound desc = could not find container \"fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6\": container with ID starting with fa15803e2bd7d778bb9d653d2196b411ed66fc078b6b8049f19471de2d42daf6 not found: ID does not exist"
Oct 31 16:44:02.485639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.485583    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:44:02.531291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:02.531255    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ndttp"]
Oct 31 16:44:02.944148 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:44:02.944114    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:44:03.050544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:44:03.050507    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" path="/var/lib/kubelet/pods/84a0df2e-35fe-4c53-90a8-fb3e18d60366/volumes"
Oct 31 16:44:32.942584 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:44:32.942548    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:45:02.943993 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:45:02.943952    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:45:32.944223 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:45:32.944185    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:46:02.945148 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:46:02.945103    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:46:08.037491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:08.037461    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:46:23.057696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.057653    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" podUID="73fdab20-9d9d-4a16-b5d0-277767873c2b" containerName="main" containerID="cri-o://45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1" gracePeriod=45
Oct 31 16:46:23.315488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.315429    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:46:23.459552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459524    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.459552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459554    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.459820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459609    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.459820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459647    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.459938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459858    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\" (UniqueName: \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.459938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459908    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:23.459938 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459916    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:23.460101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.459963    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\") pod \"73fdab20-9d9d-4a16-b5d0-277767873c2b\" (UID: \"73fdab20-9d9d-4a16-b5d0-277767873c2b\") "
Oct 31 16:46:23.460341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.460320    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:23.460444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.460428    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:23.460501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.460451    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:23.460501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.460464    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:23.461920 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.461888    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:23.462192 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.462163    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:23.462377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.462356    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56" pod "73fdab20-9d9d-4a16-b5d0-277767873c2b" (UID: "73fdab20-9d9d-4a16-b5d0-277767873c2b"). InnerVolumeSpecName "kube-api-access-t9t56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:46:23.561239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.561202    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73fdab20-9d9d-4a16-b5d0-277767873c2b-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:23.561239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.561234    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73fdab20-9d9d-4a16-b5d0-277767873c2b-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:23.561445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:23.561248    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9t56\" (UniqueName: \"kubernetes.io/projected/73fdab20-9d9d-4a16-b5d0-277767873c2b-kube-api-access-t9t56\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:24.078244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.078201    4417 generic.go:356] "Generic (PLEG): container finished" podID="73fdab20-9d9d-4a16-b5d0-277767873c2b" containerID="45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1" exitCode=0
Oct 31 16:46:24.078759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.078258    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"
Oct 31 16:46:24.078759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.078281    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" event={"ID":"73fdab20-9d9d-4a16-b5d0-277767873c2b","Type":"ContainerDied","Data":"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"}
Oct 31 16:46:24.078759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.078328    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz" event={"ID":"73fdab20-9d9d-4a16-b5d0-277767873c2b","Type":"ContainerDied","Data":"762af4b9db3ffb8411dd6181e37b73612db16db8165b054789c89cf23fbbb378"}
Oct 31 16:46:24.078759 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.078348    4417 scope.go:122] "RemoveContainer" containerID="45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"
Oct 31 16:46:24.090734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.090678    4417 scope.go:122] "RemoveContainer" containerID="45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"
Oct 31 16:46:24.091172 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:46:24.091153    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1\": container with ID starting with 45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1 not found: ID does not exist" containerID="45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"
Oct 31 16:46:24.091275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.091178    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1"} err="failed to get container status \"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1\": rpc error: code = NotFound desc = could not find container \"45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1\": container with ID starting with 45b5cde97b902a293848d4b9445d10984d700635fe504207f91b2f5d07ae19d1 not found: ID does not exist"
Oct 31 16:46:24.232020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.231993    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:46:24.278959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:24.278926    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-05db63/test-model-05db63-kserve-d85c46679-z8vzz"]
Oct 31 16:46:25.063594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:25.063559    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fdab20-9d9d-4a16-b5d0-277767873c2b" path="/var/lib/kubelet/pods/73fdab20-9d9d-4a16-b5d0-277767873c2b/volumes"
Oct 31 16:46:31.596173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:31.596138    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:46:32.943765 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:46:32.943723    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:46:43.533334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:43.533294    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:46:43.632720 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:43.632672    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:46:46.615893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.615853    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" podUID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" containerName="main" containerID="cri-o://ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0" gracePeriod=45
Oct 31 16:46:46.875861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.875770    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:46:46.894156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894120    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894171    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894284 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894230    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894362 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894333    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\" (UniqueName: \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894375    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:46.894439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894416    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894444    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\") pod \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\" (UID: \"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e\") "
Oct 31 16:46:46.894528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894480    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:46.894578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894557    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:46.894875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894852    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:46.894875 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894872    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:46.895077 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.894883    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:46.896714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.896687    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "kube-api-access-qrxsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:46:46.896714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.896699    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:46.896856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.896704    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm" pod "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" (UID: "5aa7f72a-50a1-43ee-a0a3-e02797b09c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:46:46.996439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.996406    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrxsj\" (UniqueName: \"kubernetes.io/projected/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-kube-api-access-qrxsj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:46.996439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.996438    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:46.996678 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:46.996459    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:47.200021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.199930    4417 generic.go:356] "Generic (PLEG): container finished" podID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" containerID="ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0" exitCode=0
Oct 31 16:46:47.200021 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.199973    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"
Oct 31 16:46:47.200256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.200016    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" event={"ID":"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e","Type":"ContainerDied","Data":"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"}
Oct 31 16:46:47.200256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.200063    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh" event={"ID":"5aa7f72a-50a1-43ee-a0a3-e02797b09c8e","Type":"ContainerDied","Data":"3cc581f10c51e917ce073b601288f77625c28c7f5fbe0b269b8cf7a37c7b07fd"}
Oct 31 16:46:47.200256 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.200081    4417 scope.go:122] "RemoveContainer" containerID="ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"
Oct 31 16:46:47.208743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.208723    4417 scope.go:122] "RemoveContainer" containerID="ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"
Oct 31 16:46:47.209029 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:46:47.209002    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0\": container with ID starting with ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0 not found: ID does not exist" containerID="ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"
Oct 31 16:46:47.209069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.209041    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0"} err="failed to get container status \"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0\": rpc error: code = NotFound desc = could not find container \"ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0\": container with ID starting with ad6b643dc065bca344113c31a3ea1f7a2166664c10358157603f9a21d7d7a5a0 not found: ID does not exist"
Oct 31 16:46:47.350233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.350193    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:46:47.352864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:47.352837    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-66db1c/test-model-66db1c-kserve-7cf9f787b7-c4gwh"]
Oct 31 16:46:48.534569 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:48.534521    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing" containerID="cri-o://f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de" gracePeriod=25
Oct 31 16:46:48.633381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:48.633320    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing" containerID="cri-o://d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356" gracePeriod=25
Oct 31 16:46:49.050612 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:49.050574    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" path="/var/lib/kubelet/pods/5aa7f72a-50a1-43ee-a0a3-e02797b09c8e/volumes"
Oct 31 16:46:56.111937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:56.111896    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:46:56.112334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:56.111940    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:46:57.008644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.008599    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:46:57.008831 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.008807    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" podUID="2e0be038-7784-429a-95cb-fbdbcf236917" containerName="maas-api" containerID="cri-o://f33429eec0b7b29aec64c9c24b46fa8ca809b27447ce04fdcd93895c4c652230" gracePeriod=30
Oct 31 16:46:57.257985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.257960    4417 generic.go:356] "Generic (PLEG): container finished" podID="2e0be038-7784-429a-95cb-fbdbcf236917" containerID="f33429eec0b7b29aec64c9c24b46fa8ca809b27447ce04fdcd93895c4c652230" exitCode=0
Oct 31 16:46:57.258313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.258035    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" event={"ID":"2e0be038-7784-429a-95cb-fbdbcf236917","Type":"ContainerDied","Data":"f33429eec0b7b29aec64c9c24b46fa8ca809b27447ce04fdcd93895c4c652230"}
Oct 31 16:46:57.270216 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.270193    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:46:57.407168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.407137    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\") pod \"2e0be038-7784-429a-95cb-fbdbcf236917\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") "
Oct 31 16:46:57.407371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.407194    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") pod \"2e0be038-7784-429a-95cb-fbdbcf236917\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") "
Oct 31 16:46:57.407371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.407277    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\") pod \"2e0be038-7784-429a-95cb-fbdbcf236917\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") "
Oct 31 16:46:57.407568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.407546    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\" (UniqueName: \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\") pod \"2e0be038-7784-429a-95cb-fbdbcf236917\" (UID: \"2e0be038-7784-429a-95cb-fbdbcf236917\") "
Oct 31 16:46:57.408042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.407996    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca" pod "2e0be038-7784-429a-95cb-fbdbcf236917" (UID: "2e0be038-7784-429a-95cb-fbdbcf236917"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:46:57.409749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.409716    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls" pod "2e0be038-7784-429a-95cb-fbdbcf236917" (UID: "2e0be038-7784-429a-95cb-fbdbcf236917"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:57.409946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.409920    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb" pod "2e0be038-7784-429a-95cb-fbdbcf236917" (UID: "2e0be038-7784-429a-95cb-fbdbcf236917"). InnerVolumeSpecName "kube-api-access-nf9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:46:57.410144 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.410129    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs" pod "2e0be038-7784-429a-95cb-fbdbcf236917" (UID: "2e0be038-7784-429a-95cb-fbdbcf236917"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:46:57.509297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.509260    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:57.509297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.509292    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e0be038-7784-429a-95cb-fbdbcf236917-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:57.509297 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.509304    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9zb\" (UniqueName: \"kubernetes.io/projected/2e0be038-7784-429a-95cb-fbdbcf236917-kube-api-access-nf9zb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:57.509545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:57.509313    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e0be038-7784-429a-95cb-fbdbcf236917-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:46:58.264413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:58.264386    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"
Oct 31 16:46:58.264905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:58.264384    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r" event={"ID":"2e0be038-7784-429a-95cb-fbdbcf236917","Type":"ContainerDied","Data":"0c8339edef6b24ee4cdcf36d6515e4f17dedd2135d3bf2985b454972d84e227a"}
Oct 31 16:46:58.264905 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:58.264518    4417 scope.go:122] "RemoveContainer" containerID="f33429eec0b7b29aec64c9c24b46fa8ca809b27447ce04fdcd93895c4c652230"
Oct 31 16:46:58.416501 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:58.416469    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:46:58.419499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:58.419474    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05db63-8685fcc46-t2b6r"]
Oct 31 16:46:59.054054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:46:59.054016    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0be038-7784-429a-95cb-fbdbcf236917" path="/var/lib/kubelet/pods/2e0be038-7784-429a-95cb-fbdbcf236917/volumes"
Oct 31 16:47:02.945094 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:02.945055    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:47:05.937404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:05.937367    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:47:05.937870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:05.937643    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy" containerID="cri-o://00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4" gracePeriod=30
Oct 31 16:47:06.111668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:06.111632    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:06.111668 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:06.111663    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:08.798664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.798581    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:47:08.880487 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.880465    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:47:08.936747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.936705    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\") pod \"0168c750-b6e1-415d-8118-906952bd613d\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") "
Oct 31 16:47:08.936979 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.936776    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\" (UniqueName: \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\") pod \"0168c750-b6e1-415d-8118-906952bd613d\" (UID: \"0168c750-b6e1-415d-8118-906952bd613d\") "
Oct 31 16:47:08.937095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.937070    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume" pod "0168c750-b6e1-415d-8118-906952bd613d" (UID: "0168c750-b6e1-415d-8118-906952bd613d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:08.937187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.937172    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0168c750-b6e1-415d-8118-906952bd613d-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:08.938870 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:08.938846    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc" pod "0168c750-b6e1-415d-8118-906952bd613d" (UID: "0168c750-b6e1-415d-8118-906952bd613d"). InnerVolumeSpecName "kube-api-access-sv2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:09.037558 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.037523    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\" (UniqueName: \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\") pod \"8623aa8e-31c6-4588-bf69-42772ad070b5\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") "
Oct 31 16:47:09.038177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.037696    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\") pod \"8623aa8e-31c6-4588-bf69-42772ad070b5\" (UID: \"8623aa8e-31c6-4588-bf69-42772ad070b5\") "
Oct 31 16:47:09.038177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.038025    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2nc\" (UniqueName: \"kubernetes.io/projected/0168c750-b6e1-415d-8118-906952bd613d-kube-api-access-sv2nc\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:09.038177 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.038060    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume" pod "8623aa8e-31c6-4588-bf69-42772ad070b5" (UID: "8623aa8e-31c6-4588-bf69-42772ad070b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:09.039712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.039688    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn" pod "8623aa8e-31c6-4588-bf69-42772ad070b5" (UID: "8623aa8e-31c6-4588-bf69-42772ad070b5"). InnerVolumeSpecName "kube-api-access-fttqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:09.139553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.139516    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8623aa8e-31c6-4588-bf69-42772ad070b5-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:09.139553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.139545    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fttqn\" (UniqueName: \"kubernetes.io/projected/8623aa8e-31c6-4588-bf69-42772ad070b5-kube-api-access-fttqn\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:09.324022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.323925    4417 generic.go:356] "Generic (PLEG): container finished" podID="0168c750-b6e1-415d-8118-906952bd613d" containerID="f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de" exitCode=0
Oct 31 16:47:09.324022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.323983    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"
Oct 31 16:47:09.324257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.324019    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" event={"ID":"0168c750-b6e1-415d-8118-906952bd613d","Type":"ContainerDied","Data":"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"}
Oct 31 16:47:09.324257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.324079    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz" event={"ID":"0168c750-b6e1-415d-8118-906952bd613d","Type":"ContainerDied","Data":"1b8b9ab654ee54451231f73ee07ff3f2340d86641b8511b50efb1016e8586240"}
Oct 31 16:47:09.324257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.324099    4417 scope.go:122] "RemoveContainer" containerID="f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"
Oct 31 16:47:09.325566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.325543    4417 generic.go:356] "Generic (PLEG): container finished" podID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerID="d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356" exitCode=0
Oct 31 16:47:09.325732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.325588    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" event={"ID":"8623aa8e-31c6-4588-bf69-42772ad070b5","Type":"ContainerDied","Data":"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"}
Oct 31 16:47:09.325732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.325597    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"
Oct 31 16:47:09.325732 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.325648    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj" event={"ID":"8623aa8e-31c6-4588-bf69-42772ad070b5","Type":"ContainerDied","Data":"4ca42df58aceac8489719efc9bf63031a12aa9087e10d9dda8543a98980661fe"}
Oct 31 16:47:09.333942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.333926    4417 scope.go:122] "RemoveContainer" containerID="f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"
Oct 31 16:47:09.334221 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:09.334199    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de\": container with ID starting with f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de not found: ID does not exist" containerID="f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"
Oct 31 16:47:09.334312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.334233    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de"} err="failed to get container status \"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de\": rpc error: code = NotFound desc = could not find container \"f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de\": container with ID starting with f566a605a8ffa9e8ca494207bf323ccaf68d671b9596db613984294ca31a96de not found: ID does not exist"
Oct 31 16:47:09.334312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.334258    4417 scope.go:122] "RemoveContainer" containerID="d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"
Oct 31 16:47:09.343213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.343192    4417 scope.go:122] "RemoveContainer" containerID="d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"
Oct 31 16:47:09.343493 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:09.343466    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356\": container with ID starting with d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356 not found: ID does not exist" containerID="d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"
Oct 31 16:47:09.343553 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.343501    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356"} err="failed to get container status \"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356\": rpc error: code = NotFound desc = could not find container \"d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356\": container with ID starting with d3c6ecfbfec6a0a35f583e921933b4d449bca1d3c4931d3389ca72ccb2642356 not found: ID does not exist"
Oct 31 16:47:09.475032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.475000    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:47:09.476937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.476908    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05db63-68cc6dc67f-9b9fj"]
Oct 31 16:47:09.621890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.621786    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:47:09.623965 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:09.623935    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05db63-6ddf9bc6f7-llqzz"]
Oct 31 16:47:11.050271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.050234    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0168c750-b6e1-415d-8118-906952bd613d" path="/var/lib/kubelet/pods/0168c750-b6e1-415d-8118-906952bd613d/volumes"
Oct 31 16:47:11.050768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.050753    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" path="/var/lib/kubelet/pods/8623aa8e-31c6-4588-bf69-42772ad070b5/volumes"
Oct 31 16:47:11.190764 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.190737    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:47:11.261538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261453    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261493    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261538 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261538    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261563    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261590    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261659    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261681    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.261864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261711    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.262149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261862    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:11.262149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.261974    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:11.262149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.262001    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:11.262149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.262110    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.262149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.262139    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.262417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.262154    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.262417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.262339    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:11.263958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.263923    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:47:11.263958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.263929    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:11.264206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.264183    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "kube-api-access-l7mhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:11.264206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.264194    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:11.338941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.338905    4417 generic.go:356] "Generic (PLEG): container finished" podID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerID="00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4" exitCode=0
Oct 31 16:47:11.339136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.338956    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"
Oct 31 16:47:11.339136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.338987    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerDied","Data":"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"}
Oct 31 16:47:11.339136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.339044    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9" event={"ID":"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a","Type":"ContainerDied","Data":"a3544b373305936c1cf0a6368dc42512faf7e8f36781f88c6f4f4aeee1be1a22"}
Oct 31 16:47:11.339136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.339077    4417 scope.go:122] "RemoveContainer" containerID="00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"
Oct 31 16:47:11.348179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.348157    4417 scope.go:122] "RemoveContainer" containerID="a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"
Oct 31 16:47:11.357774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.357754    4417 scope.go:122] "RemoveContainer" containerID="00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"
Oct 31 16:47:11.358137 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:11.358107    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4\": container with ID starting with 00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4 not found: ID does not exist" containerID="00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"
Oct 31 16:47:11.358257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.358151    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4"} err="failed to get container status \"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4\": rpc error: code = NotFound desc = could not find container \"00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4\": container with ID starting with 00cea42953eb94bb5e0a2e7d722da1932d0347768873927e1532561de485e5a4 not found: ID does not exist"
Oct 31 16:47:11.358257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.358178    4417 scope.go:122] "RemoveContainer" containerID="a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"
Oct 31 16:47:11.358490 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:11.358467    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be\": container with ID starting with a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be not found: ID does not exist" containerID="a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"
Oct 31 16:47:11.358564 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.358497    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be"} err="failed to get container status \"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be\": rpc error: code = NotFound desc = could not find container \"a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be\": container with ID starting with a585ef052a475f062390f67939125167fd4431aba21d5834b1254eae7dffe8be not found: ID does not exist"
Oct 31 16:47:11.362967 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.362945    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\") pod \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\" (UID: \"c2f40a63-3afb-456c-a3d6-0ad3d9296e3a\") "
Oct 31 16:47:11.363202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363183    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data" pod "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" (UID: "c2f40a63-3afb-456c-a3d6-0ad3d9296e3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:11.363303 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363289    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.363338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363306    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.363338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363318    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.363338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363326    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7mhh\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-kube-api-access-l7mhh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.363338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363336    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.363470 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.363344    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:11.750717 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.750685    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:47:11.754982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:11.754953    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-05db63-openshift-default-8cdbcbbdc-2r9c9"]
Oct 31 16:47:13.048382 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:13.048346    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" path="/var/lib/kubelet/pods/c2f40a63-3afb-456c-a3d6-0ad3d9296e3a/volumes"
Oct 31 16:47:18.162464 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:18.162430    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:47:18.261633 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:18.261595    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:47:23.163414 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:23.163364    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing" containerID="cri-o://02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1" gracePeriod=25
Oct 31 16:47:23.262167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:23.262119    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing" containerID="cri-o://7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2" gracePeriod=25
Oct 31 16:47:27.770515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:27.770473    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:27.770942 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:27.770472    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:28.597236 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.597199    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:47:28.597450 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.597428    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" podUID="353b38eb-f642-4675-8809-bfc8d6ead71b" containerName="maas-api" containerID="cri-o://c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5" gracePeriod=30
Oct 31 16:47:28.857435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.857370    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:47:28.941541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.941504    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\") pod \"353b38eb-f642-4675-8809-bfc8d6ead71b\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") "
Oct 31 16:47:28.941739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.941564    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\" (UniqueName: \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\") pod \"353b38eb-f642-4675-8809-bfc8d6ead71b\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") "
Oct 31 16:47:28.941739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.941651    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") pod \"353b38eb-f642-4675-8809-bfc8d6ead71b\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") "
Oct 31 16:47:28.941867 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.941752    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\") pod \"353b38eb-f642-4675-8809-bfc8d6ead71b\" (UID: \"353b38eb-f642-4675-8809-bfc8d6ead71b\") "
Oct 31 16:47:28.942082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.942052    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca" pod "353b38eb-f642-4675-8809-bfc8d6ead71b" (UID: "353b38eb-f642-4675-8809-bfc8d6ead71b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:28.943994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.943966    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls" pod "353b38eb-f642-4675-8809-bfc8d6ead71b" (UID: "353b38eb-f642-4675-8809-bfc8d6ead71b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:47:28.944101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.944003    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs" pod "353b38eb-f642-4675-8809-bfc8d6ead71b" (UID: "353b38eb-f642-4675-8809-bfc8d6ead71b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:47:28.944101 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:28.944036    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v" pod "353b38eb-f642-4675-8809-bfc8d6ead71b" (UID: "353b38eb-f642-4675-8809-bfc8d6ead71b"). InnerVolumeSpecName "kube-api-access-v584v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:29.042466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.042437    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:29.042466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.042462    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/353b38eb-f642-4675-8809-bfc8d6ead71b-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:29.042466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.042472    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v584v\" (UniqueName: \"kubernetes.io/projected/353b38eb-f642-4675-8809-bfc8d6ead71b-kube-api-access-v584v\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:29.042746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.042483    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/353b38eb-f642-4675-8809-bfc8d6ead71b-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:29.430269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.430230    4417 generic.go:356] "Generic (PLEG): container finished" podID="353b38eb-f642-4675-8809-bfc8d6ead71b" containerID="c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5" exitCode=0
Oct 31 16:47:29.430488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.430286    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"
Oct 31 16:47:29.430488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.430319    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" event={"ID":"353b38eb-f642-4675-8809-bfc8d6ead71b","Type":"ContainerDied","Data":"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"}
Oct 31 16:47:29.430488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.430379    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm" event={"ID":"353b38eb-f642-4675-8809-bfc8d6ead71b","Type":"ContainerDied","Data":"7912b59def6de1a749dc3a66d80665bb205734926e77a447ed9fd84985c32878"}
Oct 31 16:47:29.430488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.430405    4417 scope.go:122] "RemoveContainer" containerID="c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"
Oct 31 16:47:29.439370 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.439348    4417 scope.go:122] "RemoveContainer" containerID="c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"
Oct 31 16:47:29.439666 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:29.439643    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5\": container with ID starting with c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5 not found: ID does not exist" containerID="c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"
Oct 31 16:47:29.439745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.439678    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5"} err="failed to get container status \"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5\": rpc error: code = NotFound desc = could not find container \"c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5\": container with ID starting with c63af480119b488d62546d3119a92c9496904ad8870e949bfea278d8798c9ad5 not found: ID does not exist"
Oct 31 16:47:29.582252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.582207    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:47:29.584526 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:29.584490    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-66db1c-7fd9d88677-xndgm"]
Oct 31 16:47:31.048493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:31.048458    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353b38eb-f642-4675-8809-bfc8d6ead71b" path="/var/lib/kubelet/pods/353b38eb-f642-4675-8809-bfc8d6ead71b/volumes"
Oct 31 16:47:32.943606 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:32.943570    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:47:37.243181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:37.243148    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:47:37.243582 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:37.243372    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy" containerID="cri-o://d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722" gracePeriod=30
Oct 31 16:47:37.770441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:37.770407    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:37.770637 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:37.770442    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:47:42.497075 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.497051    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:47:42.504721 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.504698    4417 generic.go:356] "Generic (PLEG): container finished" podID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerID="d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722" exitCode=0
Oct 31 16:47:42.504848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.504740    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerDied","Data":"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"}
Oct 31 16:47:42.504848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.504790    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w" event={"ID":"cb438b8f-863e-4720-b14e-6d4bae5f151b","Type":"ContainerDied","Data":"14bbe7fcdc588188626fa5bb12b4ba0bcc2980c94a53fba9f827a6a5fa2c2dce"}
Oct 31 16:47:42.504848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.504744    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"
Oct 31 16:47:42.504966 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.504819    4417 scope.go:122] "RemoveContainer" containerID="d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"
Oct 31 16:47:42.512951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.512933    4417 scope.go:122] "RemoveContainer" containerID="15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"
Oct 31 16:47:42.520504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.520488    4417 scope.go:122] "RemoveContainer" containerID="d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"
Oct 31 16:47:42.520766 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:42.520748    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722\": container with ID starting with d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722 not found: ID does not exist" containerID="d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"
Oct 31 16:47:42.520821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.520776    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722"} err="failed to get container status \"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722\": rpc error: code = NotFound desc = could not find container \"d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722\": container with ID starting with d0faa3b1a6d38901cf33bf82962ef9034686188d1540b66cf3a70a4f7c3b3722 not found: ID does not exist"
Oct 31 16:47:42.520821 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.520797    4417 scope.go:122] "RemoveContainer" containerID="15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"
Oct 31 16:47:42.521028 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:42.521007    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8\": container with ID starting with 15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8 not found: ID does not exist" containerID="15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"
Oct 31 16:47:42.521074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.521035    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8"} err="failed to get container status \"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8\": rpc error: code = NotFound desc = could not find container \"15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8\": container with ID starting with 15bbf7463ee935da242f8084ef1e435c9f11bb6b8f48fb9f25de0964f13fafe8 not found: ID does not exist"
Oct 31 16:47:42.579683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579661    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579699    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579778 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579737    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579818    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579855    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579894    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.579940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579929    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.580055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.579980    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.580055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580022    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\") pod \"cb438b8f-863e-4720-b14e-6d4bae5f151b\" (UID: \"cb438b8f-863e-4720-b14e-6d4bae5f151b\") "
Oct 31 16:47:42.580168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580080    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:42.580168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580146    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:42.580283 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580175    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:42.580381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580340    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:42.580524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580498    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:42.580609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580530    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.580609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580551    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.580609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580566    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.580609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.580580    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.581890 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.581860    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:42.581993 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.581969    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:47:42.582049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.582021    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:42.582209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.582184    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql" pod "cb438b8f-863e-4720-b14e-6d4bae5f151b" (UID: "cb438b8f-863e-4720-b14e-6d4bae5f151b"). InnerVolumeSpecName "kube-api-access-x46ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:42.681240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.681211    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.681240 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.681237    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46ql\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-kube-api-access-x46ql\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.681456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.681249    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb438b8f-863e-4720-b14e-6d4bae5f151b-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.681456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.681259    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.681456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.681269    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb438b8f-863e-4720-b14e-6d4bae5f151b-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:42.698241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.698207    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:47:42.916695 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.916665    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:47:42.920119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:42.920086    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-66db1c-openshift-default-798cbbdd45-5ds9w"]
Oct 31 16:47:43.049042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.049000    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" path="/var/lib/kubelet/pods/cb438b8f-863e-4720-b14e-6d4bae5f151b/volumes"
Oct 31 16:47:43.418664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.415926    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:47:43.510440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510408    4417 generic.go:356] "Generic (PLEG): container finished" podID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerID="02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1" exitCode=0
Oct 31 16:47:43.510854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510459    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"
Oct 31 16:47:43.510854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510483    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" event={"ID":"26eabd75-ed00-4313-8f44-ebe7f6faf756","Type":"ContainerDied","Data":"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"}
Oct 31 16:47:43.510854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510544    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6" event={"ID":"26eabd75-ed00-4313-8f44-ebe7f6faf756","Type":"ContainerDied","Data":"30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e"}
Oct 31 16:47:43.510854 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510574    4417 scope.go:122] "RemoveContainer" containerID="02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"
Oct 31 16:47:43.511046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.510958    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:47:43.512723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.512702    4417 generic.go:356] "Generic (PLEG): container finished" podID="308e9eb2-2978-440c-920f-a6950276037d" containerID="7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2" exitCode=0
Oct 31 16:47:43.512844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.512747    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"
Oct 31 16:47:43.512844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.512763    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" event={"ID":"308e9eb2-2978-440c-920f-a6950276037d","Type":"ContainerDied","Data":"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"}
Oct 31 16:47:43.512844 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.512796    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs" event={"ID":"308e9eb2-2978-440c-920f-a6950276037d","Type":"ContainerDied","Data":"9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672"}
Oct 31 16:47:43.519032 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.519009    4417 scope.go:122] "RemoveContainer" containerID="02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"
Oct 31 16:47:43.519312 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:43.519285    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1\": container with ID starting with 02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1 not found: ID does not exist" containerID="02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"
Oct 31 16:47:43.519387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.519321    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1"} err="failed to get container status \"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1\": rpc error: code = NotFound desc = could not find container \"02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1\": container with ID starting with 02ede3aa36b674a3a906da0bf0283bae7f00c40db05863adc6d14206c30cfef1 not found: ID does not exist"
Oct 31 16:47:43.519387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.519338    4417 scope.go:122] "RemoveContainer" containerID="7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"
Oct 31 16:47:43.527474 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.527456    4417 scope.go:122] "RemoveContainer" containerID="7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"
Oct 31 16:47:43.527769 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:43.527749    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2\": container with ID starting with 7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2 not found: ID does not exist" containerID="7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"
Oct 31 16:47:43.527830 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.527776    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2"} err="failed to get container status \"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2\": rpc error: code = NotFound desc = could not find container \"7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2\": container with ID starting with 7dcd019ebe0d86ad73a08f1bb4289f3523b1d2a8bbc4d355bf04fdecc69c28e2 not found: ID does not exist"
Oct 31 16:47:43.591697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.591659    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\") pod \"26eabd75-ed00-4313-8f44-ebe7f6faf756\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") "
Oct 31 16:47:43.591906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.591774    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\" (UniqueName: \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\") pod \"26eabd75-ed00-4313-8f44-ebe7f6faf756\" (UID: \"26eabd75-ed00-4313-8f44-ebe7f6faf756\") "
Oct 31 16:47:43.592097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.592066    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume" pod "26eabd75-ed00-4313-8f44-ebe7f6faf756" (UID: "26eabd75-ed00-4313-8f44-ebe7f6faf756"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:43.593879 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.593847    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth" pod "26eabd75-ed00-4313-8f44-ebe7f6faf756" (UID: "26eabd75-ed00-4313-8f44-ebe7f6faf756"). InnerVolumeSpecName "kube-api-access-hxjth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:43.693511 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693403    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\") pod \"308e9eb2-2978-440c-920f-a6950276037d\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") "
Oct 31 16:47:43.693743 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693535    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\" (UniqueName: \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\") pod \"308e9eb2-2978-440c-920f-a6950276037d\" (UID: \"308e9eb2-2978-440c-920f-a6950276037d\") "
Oct 31 16:47:43.693837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693815    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume" pod "308e9eb2-2978-440c-920f-a6950276037d" (UID: "308e9eb2-2978-440c-920f-a6950276037d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:47:43.693950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693929    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26eabd75-ed00-4313-8f44-ebe7f6faf756-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:43.693950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693948    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308e9eb2-2978-440c-920f-a6950276037d-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:43.694103 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.693958    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjth\" (UniqueName: \"kubernetes.io/projected/26eabd75-ed00-4313-8f44-ebe7f6faf756-kube-api-access-hxjth\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:43.695585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.695555    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5" pod "308e9eb2-2978-440c-920f-a6950276037d" (UID: "308e9eb2-2978-440c-920f-a6950276037d"). InnerVolumeSpecName "kube-api-access-k7vs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:43.794591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.794553    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vs5\" (UniqueName: \"kubernetes.io/projected/308e9eb2-2978-440c-920f-a6950276037d-kube-api-access-k7vs5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:43.851701 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:43.851661    4417 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308e9eb2_2978_440c_920f_a6950276037d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eabd75_ed00_4313_8f44_ebe7f6faf756.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26eabd75_ed00_4313_8f44_ebe7f6faf756.slice/crio-30bc09286b0ff642425a93b61c5f1a332b4989a24f6cfbabaac1fdb0644bd19e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308e9eb2_2978_440c_920f_a6950276037d.slice/crio-9228cd1f91af9cc8940d72927f4ef7179a6dc3ca29bce234452c95c7c0555672\": RecentStats: unable to find data in memory cache]"
Oct 31 16:47:43.926638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.926575    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:47:43.930111 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:43.930082    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-66db1c-d95868c6c-npsqs"]
Oct 31 16:47:44.031945 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:44.031861    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:47:44.039466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:44.039442    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-66db1c-77f86d9bbc-7rsl6"]
Oct 31 16:47:45.047261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:45.047230    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" path="/var/lib/kubelet/pods/26eabd75-ed00-4313-8f44-ebe7f6faf756/volumes"
Oct 31 16:47:45.047698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:45.047607    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308e9eb2-2978-440c-920f-a6950276037d" path="/var/lib/kubelet/pods/308e9eb2-2978-440c-920f-a6950276037d/volumes"
Oct 31 16:47:57.717840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:57.717798    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main" containerID="cri-o://7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2" gracePeriod=45
Oct 31 16:47:57.967594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:57.967569    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:47:58.024948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.024856    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.024948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.024892    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.024948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.024921    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.025286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.024970    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\" (UniqueName: \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.025286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025071    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.025286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025126    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\") pod \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\" (UID: \"5aac4bdf-d175-4447-ae7c-3b991ac5703e\") "
Oct 31 16:47:58.025286 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025174    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.025521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025361    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.025521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025453    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.025521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.025500    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.027241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.027212    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:47:58.027347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.027213    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:47:58.027347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.027258    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8" pod "5aac4bdf-d175-4447-ae7c-3b991ac5703e" (UID: "5aac4bdf-d175-4447-ae7c-3b991ac5703e"). InnerVolumeSpecName "kube-api-access-sfhj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:47:58.126419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.126385    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.126419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.126415    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aac4bdf-d175-4447-ae7c-3b991ac5703e-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.126419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.126426    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfhj8\" (UniqueName: \"kubernetes.io/projected/5aac4bdf-d175-4447-ae7c-3b991ac5703e-kube-api-access-sfhj8\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.126706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.126436    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.126706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.126447    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aac4bdf-d175-4447-ae7c-3b991ac5703e-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:47:58.584463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.584431    4417 generic.go:356] "Generic (PLEG): container finished" podID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerID="7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2" exitCode=0
Oct 31 16:47:58.584463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.584466    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" event={"ID":"5aac4bdf-d175-4447-ae7c-3b991ac5703e","Type":"ContainerDied","Data":"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"}
Oct 31 16:47:58.584744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.584471    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"
Oct 31 16:47:58.584744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.584493    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w" event={"ID":"5aac4bdf-d175-4447-ae7c-3b991ac5703e","Type":"ContainerDied","Data":"4e25383e677215ab6c7cbf795f0ba6cd5d60410a3fc628eb2b697aa2197f2228"}
Oct 31 16:47:58.584744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.584516    4417 scope.go:122] "RemoveContainer" containerID="7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"
Oct 31 16:47:58.593233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.593217    4417 scope.go:122] "RemoveContainer" containerID="7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"
Oct 31 16:47:58.593497 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:47:58.593477    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2\": container with ID starting with 7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2 not found: ID does not exist" containerID="7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"
Oct 31 16:47:58.593544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.593504    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2"} err="failed to get container status \"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2\": rpc error: code = NotFound desc = could not find container \"7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2\": container with ID starting with 7e6a6e37964b9ee27bbc851e48fe8feb743e2680e2d44b8862dff974397bcdc2 not found: ID does not exist"
Oct 31 16:47:58.734977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.734945    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:47:58.780436 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:58.780406    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/distinct-w3-f7821a-kserve-57b94d7fc7-swc9w"]
Oct 31 16:47:59.049450 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:47:59.049414    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" path="/var/lib/kubelet/pods/5aac4bdf-d175-4447-ae7c-3b991ac5703e/volumes"
Oct 31 16:48:02.943306 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:02.943271    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:48:17.681070 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.680992    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:48:17.681611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.681187    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy" containerID="cri-o://10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9" gracePeriod=30
Oct 31 16:48:17.726803 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.726767    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:48:17.726959 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.726934    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" containerID="cri-o://5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2" gracePeriod=30
Oct 31 16:48:17.728276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.728251    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:48:17.728444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.728423    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy" containerID="cri-o://e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df" gracePeriod=30
Oct 31 16:48:17.772586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.772561    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:48:17.772747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:17.772729    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy" containerID="cri-o://1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5" gracePeriod=30
Oct 31 16:48:18.787027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:18.786989    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cnft9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:48:18.787415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:18.787055    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:48:19.870552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:19.870510    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:48:20.772193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772159    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"]
Oct 31 16:48:20.772698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772680    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:20.772698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772699    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772711    4417 cpu_manager.go:414] "containerMap: removing container" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772716    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772726    4417 cpu_manager.go:414] "containerMap: removing container" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772732    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772740    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772745    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772752    4417 cpu_manager.go:414] "containerMap: removing container" podUID="73fdab20-9d9d-4a16-b5d0-277767873c2b" containerName="main"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772756    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fdab20-9d9d-4a16-b5d0-277767873c2b" containerName="main"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772764    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772768    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772776    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772783    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772790    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772795    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772804    4417 cpu_manager.go:414] "containerMap: removing container" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerName="discovery"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772808    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerName="discovery"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772815    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772821    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772827    4417 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" containerName="main"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772831    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" containerName="main"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772837    4417 cpu_manager.go:414] "containerMap: removing container" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772841    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772848    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" containerName="revoke-keys"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772852    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" containerName="revoke-keys"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772862    4417 cpu_manager.go:414] "containerMap: removing container" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing"
Oct 31 16:48:20.772852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772867    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772881    4417 cpu_manager.go:414] "containerMap: removing container" podUID="2e0be038-7784-429a-95cb-fbdbcf236917" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772885    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0be038-7784-429a-95cb-fbdbcf236917" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772892    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772896    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772903    4417 cpu_manager.go:414] "containerMap: removing container" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772908    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772914    4417 cpu_manager.go:414] "containerMap: removing container" podUID="353b38eb-f642-4675-8809-bfc8d6ead71b" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772919    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353b38eb-f642-4675-8809-bfc8d6ead71b" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772927    4417 cpu_manager.go:414] "containerMap: removing container" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772931    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772985    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7f72a-50a1-43ee-a0a3-e02797b09c8e" containerName="main"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772993    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="4727544f-ec71-4aa1-96cd-289815522f8e" containerName="payload-pre-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.772998    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee0b0a3-9ed2-423b-aa4a-b1eb7a287cfb" containerName="revoke-keys"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773004    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="308e9eb2-2978-440c-920f-a6950276037d" containerName="payload-pre-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773010    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0be038-7784-429a-95cb-fbdbcf236917" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773017    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773026    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="26eabd75-ed00-4313-8f44-ebe7f6faf756" containerName="payload-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773035    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773042    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="353b38eb-f642-4675-8809-bfc8d6ead71b" containerName="maas-api"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773048    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aac4bdf-d175-4447-ae7c-3b991ac5703e" containerName="main"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773053    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773059    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa484d9-13c0-4a05-b493-786a96446e90" containerName="payload-pre-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773066    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01dac0c-4523-4f5a-b18a-653d540abae0" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773074    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773081    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fdab20-9d9d-4a16-b5d0-277767873c2b" containerName="main"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773090    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4c0100-b01f-4d6f-a68d-40a459879e34" containerName="payload-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773099    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8623aa8e-31c6-4588-bf69-42772ad070b5" containerName="payload-pre-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773107    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0168c750-b6e1-415d-8118-906952bd613d" containerName="payload-processing"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773115    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a0df2e-35fe-4c53-90a8-fb3e18d60366" containerName="discovery"
Oct 31 16:48:20.774073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.773120    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:20.776900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.776876    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:20.833219 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.833193    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"]
Oct 31 16:48:20.870748 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.870721    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:48:20.871124 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.870966    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:48:20.939203 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.939172    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:20.939341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.939233    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8xq\" (UniqueName: \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:20.939388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:20.939345    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.040302 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.040263    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8xq\" (UniqueName: \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.040445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.040339    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.040445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.040412    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.041034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.041012    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.042819 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.042794    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.092956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.092935    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8xq\" (UniqueName: \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\") pod \"maas-api-revoke-sub-9bcdb602-6mt8g\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.393191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.393109    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:21.767138 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:21.767054    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"]
Oct 31 16:48:21.774814 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:48:21.774791    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f8cc5f_bfc7_4c0f_9d98_b4159fc0f4d4.slice/crio-6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84 WatchSource:0}: Error finding container 6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84: Status 404 returned error can't find the container with id 6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84
Oct 31 16:48:22.692392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:22.692362    4417 generic.go:356] "Generic (PLEG): container finished" podID="53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" containerID="fc6979bcc2127b346511bee0539ffba25e667fe928f85811801eb24dfb8c1a9a" exitCode=0
Oct 31 16:48:22.692848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:22.692439    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g" event={"ID":"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4","Type":"ContainerDied","Data":"fc6979bcc2127b346511bee0539ffba25e667fe928f85811801eb24dfb8c1a9a"}
Oct 31 16:48:22.692848 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:22.692492    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g" event={"ID":"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4","Type":"ContainerStarted","Data":"6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84"}
Oct 31 16:48:22.996852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:22.996830    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:48:23.083145 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.083119    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:48:23.135518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.135494    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:48:23.138998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.138972    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:48:23.165275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165246    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165282    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165318    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165369    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165409    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165475    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165531    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165560    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165571    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165611    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165643    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\") pod \"c5900102-7c30-4633-b058-dc36fe25d45a\" (UID: \"c5900102-7c30-4633-b058-dc36fe25d45a\") "
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.165651    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.166028    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.166048    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.166065    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.166285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.166153    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.167271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.166379    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.168357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.168334    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.168357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.168345    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.168489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.168399    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.169260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.169236    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc" pod "c5900102-7c30-4633-b058-dc36fe25d45a" (UID: "c5900102-7c30-4633-b058-dc36fe25d45a"). InnerVolumeSpecName "kube-api-access-ntdzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.266387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266352    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266393    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266424    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266458    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266493    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266519    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266551    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.266591 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266580    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266609    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266677    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266726    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266734    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266757    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266765    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266795    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266829    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266873    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266876    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266920    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266949    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.267006 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.266982    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267016    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267053    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267090    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267104    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267133    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267179    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267209    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267219    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\") pod \"201b9126-7885-426a-993a-07c4d4345ca5\" (UID: \"201b9126-7885-426a-993a-07c4d4345ca5\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267314    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267430    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\") pod \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\" (UID: \"0c2d26ea-65dd-4a7c-a814-a15aefa8947f\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267469    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\") pod \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\" (UID: \"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc\") "
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267585    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267514    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267727    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267805    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.267820    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.268020    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.268346 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.268142    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.268433    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.268606    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.268834    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269428    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269449    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntdzc\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-kube-api-access-ntdzc\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269475    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269493    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269508    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269522    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269537    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5900102-7c30-4633-b058-dc36fe25d45a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269553    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269568    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269580    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/201b9126-7885-426a-993a-07c4d4345ca5-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269597    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5900102-7c30-4633-b058-dc36fe25d45a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269612    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269657    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269672    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269685    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269699    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269713    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269727    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269747    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269761    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5900102-7c30-4633-b058-dc36fe25d45a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269775    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5900102-7c30-4633-b058-dc36fe25d45a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.269789    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.270074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.270016    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.272156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.271602    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "kube-api-access-n2dnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.272156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.271765    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.272156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.271830    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.272156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.271850    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "kube-api-access-gmkb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.272379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.272169    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.273040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.272989    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.273974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.273899    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq" pod "201b9126-7885-426a-993a-07c4d4345ca5" (UID: "201b9126-7885-426a-993a-07c4d4345ca5"). InnerVolumeSpecName "kube-api-access-f6qmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.273974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.273563    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:48:23.274202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.274172    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo" pod "0c2d26ea-65dd-4a7c-a814-a15aefa8947f" (UID: "0c2d26ea-65dd-4a7c-a814-a15aefa8947f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:48:23.274313 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.274284    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token" pod "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" (UID: "d15efc78-8629-4ca3-9f40-dca9ebfb7dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.370815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370785    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.370815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370809    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.370815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370819    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6qmq\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-kube-api-access-f6qmq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370828    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370838    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/201b9126-7885-426a-993a-07c4d4345ca5-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370848    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dnj\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-kube-api-access-n2dnj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370856    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/201b9126-7885-426a-993a-07c4d4345ca5-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370864    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370873    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370883    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkb4\" (UniqueName: \"kubernetes.io/projected/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-kube-api-access-gmkb4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370893    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2d26ea-65dd-4a7c-a814-a15aefa8947f-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.371046 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.370902    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/201b9126-7885-426a-993a-07c4d4345ca5-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.698912 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.698875    4417 generic.go:356] "Generic (PLEG): container finished" podID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerID="1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5" exitCode=0
Oct 31 16:48:23.699405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.698928    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"
Oct 31 16:48:23.699405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.698967    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" event={"ID":"0c2d26ea-65dd-4a7c-a814-a15aefa8947f","Type":"ContainerDied","Data":"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"}
Oct 31 16:48:23.699405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.699026    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj" event={"ID":"0c2d26ea-65dd-4a7c-a814-a15aefa8947f","Type":"ContainerDied","Data":"92b6808d38c96d1dff5cbdd21a75e09ae8e5a24f634daaf3f8339974197b2d87"}
Oct 31 16:48:23.699405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.699051    4417 scope.go:122] "RemoveContainer" containerID="1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"
Oct 31 16:48:23.700329 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.700308    4417 generic.go:356] "Generic (PLEG): container finished" podID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerID="5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2" exitCode=0
Oct 31 16:48:23.700398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.700339    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" event={"ID":"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc","Type":"ContainerDied","Data":"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"}
Oct 31 16:48:23.700398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.700362    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"
Oct 31 16:48:23.700398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.700380    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9" event={"ID":"d15efc78-8629-4ca3-9f40-dca9ebfb7dcc","Type":"ContainerDied","Data":"4d28f1668a5b723a867e0b431202f51bda2bec5689f43ad0a96a623fc3e06cf7"}
Oct 31 16:48:23.701813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.701791    4417 generic.go:356] "Generic (PLEG): container finished" podID="201b9126-7885-426a-993a-07c4d4345ca5" containerID="e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df" exitCode=0
Oct 31 16:48:23.701955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.701818    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" event={"ID":"201b9126-7885-426a-993a-07c4d4345ca5","Type":"ContainerDied","Data":"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"}
Oct 31 16:48:23.701955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.701842    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"
Oct 31 16:48:23.701955 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.701858    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8" event={"ID":"201b9126-7885-426a-993a-07c4d4345ca5","Type":"ContainerDied","Data":"4f955c9af59690288637ce058f4a3cb105b0ad282acb9801084f517bb9f1e8db"}
Oct 31 16:48:23.703431 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.703411    4417 generic.go:356] "Generic (PLEG): container finished" podID="c5900102-7c30-4633-b058-dc36fe25d45a" containerID="10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9" exitCode=0
Oct 31 16:48:23.703535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.703465    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"
Oct 31 16:48:23.703535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.703497    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" event={"ID":"c5900102-7c30-4633-b058-dc36fe25d45a","Type":"ContainerDied","Data":"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"}
Oct 31 16:48:23.703535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.703531    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c" event={"ID":"c5900102-7c30-4633-b058-dc36fe25d45a","Type":"ContainerDied","Data":"725e6d5ec5ce3c51faf2892511eec12b0ffeafeee0337eda1c4e9693520f869c"}
Oct 31 16:48:23.710699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.710684    4417 scope.go:122] "RemoveContainer" containerID="1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"
Oct 31 16:48:23.711156 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:23.711106    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5\": container with ID starting with 1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5 not found: ID does not exist" containerID="1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"
Oct 31 16:48:23.711243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.711165    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5"} err="failed to get container status \"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5\": rpc error: code = NotFound desc = could not find container \"1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5\": container with ID starting with 1987871259f2206df61a13ae091df74978542ded6903b02346b0a6ba07cbdce5 not found: ID does not exist"
Oct 31 16:48:23.711243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.711188    4417 scope.go:122] "RemoveContainer" containerID="5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"
Oct 31 16:48:23.731954 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.731926    4417 scope.go:122] "RemoveContainer" containerID="5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"
Oct 31 16:48:23.734957 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:23.734921    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2\": container with ID starting with 5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2 not found: ID does not exist" containerID="5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"
Oct 31 16:48:23.735089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.734952    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2"} err="failed to get container status \"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2\": rpc error: code = NotFound desc = could not find container \"5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2\": container with ID starting with 5ff1ec94e81a8e681cfb793c612e1fd8326f0566f5d23130771207ab1aa6e1c2 not found: ID does not exist"
Oct 31 16:48:23.735089 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.734971    4417 scope.go:122] "RemoveContainer" containerID="e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"
Oct 31 16:48:23.760785 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.760756    4417 scope.go:122] "RemoveContainer" containerID="e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"
Oct 31 16:48:23.761347 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:23.761320    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df\": container with ID starting with e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df not found: ID does not exist" containerID="e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"
Oct 31 16:48:23.761457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.761359    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df"} err="failed to get container status \"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df\": rpc error: code = NotFound desc = could not find container \"e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df\": container with ID starting with e6d297619a4e2b29a148e7ef1601bc184a627356a708fa1d5af4b43bd748c8df not found: ID does not exist"
Oct 31 16:48:23.761457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.761384    4417 scope.go:122] "RemoveContainer" containerID="10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"
Oct 31 16:48:23.773078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.773051    4417 scope.go:122] "RemoveContainer" containerID="10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"
Oct 31 16:48:23.773356 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:23.773336    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9\": container with ID starting with 10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9 not found: ID does not exist" containerID="10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"
Oct 31 16:48:23.773443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.773365    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9"} err="failed to get container status \"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9\": rpc error: code = NotFound desc = could not find container \"10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9\": container with ID starting with 10fba3f99a1ee75681018b09b679b8d63188f6503d42d517df222d5dd8aff6e9 not found: ID does not exist"
Oct 31 16:48:23.842841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.842807    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:23.850667 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.850644    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:48:23.895347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.895318    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ghk9c"]
Oct 31 16:48:23.953119 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.953051    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:48:23.955004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.954980    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cnft9"]
Oct 31 16:48:23.975807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.975776    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\" (UniqueName: \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\") pod \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") "
Oct 31 16:48:23.975939 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.975844    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\") pod \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") "
Oct 31 16:48:23.976013 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.975977    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\") pod \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\" (UID: \"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4\") "
Oct 31 16:48:23.976229 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.976205    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca" pod "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" (UID: "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:23.976335 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.976305    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:23.977996 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.977966    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq" pod "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" (UID: "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4"). InnerVolumeSpecName "kube-api-access-6p8xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:23.978088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:23.977993    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca" pod "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" (UID: "53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:48:24.056528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.056498    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:48:24.060073 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.060041    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-mqgxj"]
Oct 31 16:48:24.076990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.076961    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:24.076990 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.076988    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8xq\" (UniqueName: \"kubernetes.io/projected/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4-kube-api-access-6p8xq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:24.159601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.159565    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:48:24.164681 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.164655    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x2fs8"]
Oct 31 16:48:24.709128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.709103    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"
Oct 31 16:48:24.709577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.709098    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g" event={"ID":"53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4","Type":"ContainerDied","Data":"6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84"}
Oct 31 16:48:24.709577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.709194    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6490703df27c517f99bc1382e06232dc5c92aed42f83bbbd482efd0499c48f84"
Oct 31 16:48:24.871157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:24.871088    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" podUID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" containerName="payload-pre-processing" containerID="cri-o://f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2" gracePeriod=25
Oct 31 16:48:25.049777 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.049744    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" path="/var/lib/kubelet/pods/0c2d26ea-65dd-4a7c-a814-a15aefa8947f/volumes"
Oct 31 16:48:25.051235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.050305    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201b9126-7885-426a-993a-07c4d4345ca5" path="/var/lib/kubelet/pods/201b9126-7885-426a-993a-07c4d4345ca5/volumes"
Oct 31 16:48:25.051235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.050774    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" path="/var/lib/kubelet/pods/c5900102-7c30-4633-b058-dc36fe25d45a/volumes"
Oct 31 16:48:25.051728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.051706    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" path="/var/lib/kubelet/pods/d15efc78-8629-4ca3-9f40-dca9ebfb7dcc/volumes"
Oct 31 16:48:25.112933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.112908    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:48:25.185585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.185559    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\") pod \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") "
Oct 31 16:48:25.185776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.185599    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\" (UniqueName: \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\") pod \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\" (UID: \"8ce917eb-f966-4e0c-aad5-7c5ace3437fa\") "
Oct 31 16:48:25.185943 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.185921    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume" pod "8ce917eb-f966-4e0c-aad5-7c5ace3437fa" (UID: "8ce917eb-f966-4e0c-aad5-7c5ace3437fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:25.187651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.187611    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7" pod "8ce917eb-f966-4e0c-aad5-7c5ace3437fa" (UID: "8ce917eb-f966-4e0c-aad5-7c5ace3437fa"). InnerVolumeSpecName "kube-api-access-jqbs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:25.286589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.286521    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:25.286589 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.286545    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbs7\" (UniqueName: \"kubernetes.io/projected/8ce917eb-f966-4e0c-aad5-7c5ace3437fa-kube-api-access-jqbs7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:25.718835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.718804    4417 generic.go:356] "Generic (PLEG): container finished" podID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" containerID="f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2" exitCode=0
Oct 31 16:48:25.718835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.718839    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" event={"ID":"8ce917eb-f966-4e0c-aad5-7c5ace3437fa","Type":"ContainerDied","Data":"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"}
Oct 31 16:48:25.719424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.718861    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"
Oct 31 16:48:25.719424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.718876    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb" event={"ID":"8ce917eb-f966-4e0c-aad5-7c5ace3437fa","Type":"ContainerDied","Data":"d66eca21a243e2ec2d51a3f7f10bc110b0e5589c9fdb8d44f0fe3b33c19ac200"}
Oct 31 16:48:25.719424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.718897    4417 scope.go:122] "RemoveContainer" containerID="f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"
Oct 31 16:48:25.727206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.727188    4417 scope.go:122] "RemoveContainer" containerID="f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"
Oct 31 16:48:25.727473 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:25.727453    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2\": container with ID starting with f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2 not found: ID does not exist" containerID="f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"
Oct 31 16:48:25.727515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.727486    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2"} err="failed to get container status \"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2\": rpc error: code = NotFound desc = could not find container \"f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2\": container with ID starting with f69639f83792a6ca3ecc4996a31cf023641caed6f0ac8c8d22a1e724c401cdf2 not found: ID does not exist"
Oct 31 16:48:25.868511 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.868482    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:48:25.915270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:25.915236    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f7821a-7c94f8c899-7xkrb"]
Oct 31 16:48:27.047000 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:27.046971    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" path="/var/lib/kubelet/pods/8ce917eb-f966-4e0c-aad5-7c5ace3437fa/volumes"
Oct 31 16:48:30.604515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.604479    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:48:30.605005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.604698    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" podUID="413a9095-e5d6-4466-a0e1-087b35f7920e" containerName="maas-api" containerID="cri-o://f8449833de6d704b40dca395d40be554ea2a0cc397b3fbe9b9bffc6ce661bcc7" gracePeriod=30
Oct 31 16:48:30.745086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.745059    4417 generic.go:356] "Generic (PLEG): container finished" podID="413a9095-e5d6-4466-a0e1-087b35f7920e" containerID="f8449833de6d704b40dca395d40be554ea2a0cc397b3fbe9b9bffc6ce661bcc7" exitCode=0
Oct 31 16:48:30.745239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.745126    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" event={"ID":"413a9095-e5d6-4466-a0e1-087b35f7920e","Type":"ContainerDied","Data":"f8449833de6d704b40dca395d40be554ea2a0cc397b3fbe9b9bffc6ce661bcc7"}
Oct 31 16:48:30.844285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.844263    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:48:30.932306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932228    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\") pod \"413a9095-e5d6-4466-a0e1-087b35f7920e\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") "
Oct 31 16:48:30.932306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932276    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\" (UniqueName: \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\") pod \"413a9095-e5d6-4466-a0e1-087b35f7920e\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") "
Oct 31 16:48:30.932306 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932304    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\") pod \"413a9095-e5d6-4466-a0e1-087b35f7920e\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") "
Oct 31 16:48:30.932568 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932486    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") pod \"413a9095-e5d6-4466-a0e1-087b35f7920e\" (UID: \"413a9095-e5d6-4466-a0e1-087b35f7920e\") "
Oct 31 16:48:30.932661 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932644    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca" pod "413a9095-e5d6-4466-a0e1-087b35f7920e" (UID: "413a9095-e5d6-4466-a0e1-087b35f7920e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:48:30.932838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.932818    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/413a9095-e5d6-4466-a0e1-087b35f7920e-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:30.934442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.934419    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs" pod "413a9095-e5d6-4466-a0e1-087b35f7920e" (UID: "413a9095-e5d6-4466-a0e1-087b35f7920e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:48:30.934575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.934456    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd" pod "413a9095-e5d6-4466-a0e1-087b35f7920e" (UID: "413a9095-e5d6-4466-a0e1-087b35f7920e"). InnerVolumeSpecName "kube-api-access-dm8jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:48:30.934575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:30.934468    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls" pod "413a9095-e5d6-4466-a0e1-087b35f7920e" (UID: "413a9095-e5d6-4466-a0e1-087b35f7920e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:48:31.033565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.033534    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:31.033565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.033560    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/413a9095-e5d6-4466-a0e1-087b35f7920e-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:31.033565 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.033570    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8jd\" (UniqueName: \"kubernetes.io/projected/413a9095-e5d6-4466-a0e1-087b35f7920e-kube-api-access-dm8jd\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:48:31.750676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.750640    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp" event={"ID":"413a9095-e5d6-4466-a0e1-087b35f7920e","Type":"ContainerDied","Data":"101d70165bb7ceb250bc761eff5c2345f36f69126f0029df96b76ca144020f85"}
Oct 31 16:48:31.750676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.750664    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"
Oct 31 16:48:31.751183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.750701    4417 scope.go:122] "RemoveContainer" containerID="f8449833de6d704b40dca395d40be554ea2a0cc397b3fbe9b9bffc6ce661bcc7"
Oct 31 16:48:31.901460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.901429    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:48:31.947505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:31.947475    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f7821a-54fcc7fb89-c6mkp"]
Oct 31 16:48:32.943984 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:48:32.943948    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:48:33.047310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:33.047279    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413a9095-e5d6-4466-a0e1-087b35f7920e" path="/var/lib/kubelet/pods/413a9095-e5d6-4466-a0e1-087b35f7920e/volumes"
Oct 31 16:48:36.486868 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:36.486836    4417 scope.go:122] "RemoveContainer" containerID="e55145a795da0d6843dd11fb540bcadff701d212472a1864150a3076a7de522f"
Oct 31 16:48:47.194857 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.194819    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195149    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195159    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6218c303-4ea9-45b7-ad43-17c0b7ee2bff" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195171    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195176    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f40a63-3afb-456c-a3d6-0ad3d9296e3a" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195186    4417 cpu_manager.go:414] "containerMap: removing container" podUID="53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" containerName="revoke-keys"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195191    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" containerName="revoke-keys"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195198    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195202    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195212    4417 cpu_manager.go:414] "containerMap: removing container" podUID="413a9095-e5d6-4466-a0e1-087b35f7920e" containerName="maas-api"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195217    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413a9095-e5d6-4466-a0e1-087b35f7920e" containerName="maas-api"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195227    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195231    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195241    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195245    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195251    4417 cpu_manager.go:414] "containerMap: removing container" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195256    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195263    4417 cpu_manager.go:414] "containerMap: removing container" podUID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" containerName="payload-pre-processing"
Oct 31 16:48:47.195268 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195268    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" containerName="payload-pre-processing"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195317    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce917eb-f966-4e0c-aad5-7c5ace3437fa" containerName="payload-pre-processing"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195329    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb438b8f-863e-4720-b14e-6d4bae5f151b" containerName="istio-proxy"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195338    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" containerName="revoke-keys"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195342    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5900102-7c30-4633-b058-dc36fe25d45a" containerName="istio-proxy"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195349    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="413a9095-e5d6-4466-a0e1-087b35f7920e" containerName="maas-api"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195354    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15efc78-8629-4ca3-9f40-dca9ebfb7dcc" containerName="istio-proxy"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195360    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2d26ea-65dd-4a7c-a814-a15aefa8947f" containerName="istio-proxy"
Oct 31 16:48:47.195914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.195366    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="201b9126-7885-426a-993a-07c4d4345ca5" containerName="istio-proxy"
Oct 31 16:48:47.199969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.199947    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.280866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.280823    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5djp\" (UniqueName: \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.280866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.280872    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.281127 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.281025    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.294291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.294254    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 16:48:47.294291 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.294275    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 16:48:47.294469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.294279    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 16:48:47.294469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.294342    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v4hcf\""
Oct 31 16:48:47.295796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.295776    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:48:47.381554 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.381508    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.381774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.381575    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5djp\" (UniqueName: \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.381774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.381631    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.384304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.384274    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.384449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.384274    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.434495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.434458    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5djp\" (UniqueName: \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\") pod \"maas-controller-6c95c64d65-ttc9p\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") " pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.512728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.512653    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:47.876575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.876541    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:48:47.884182 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:48:47.884154    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e4da77_beeb_4fd0_ac02_8aa4c7a58624.slice/crio-656511351c30c9e795c45e62ed071c0c74153ee895243cd692956100c993b51e WatchSource:0}: Error finding container 656511351c30c9e795c45e62ed071c0c74153ee895243cd692956100c993b51e: Status 404 returned error can't find the container with id 656511351c30c9e795c45e62ed071c0c74153ee895243cd692956100c993b51e
Oct 31 16:48:47.885562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:47.885536    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:48:48.827160 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:48.827125    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" event={"ID":"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624","Type":"ContainerStarted","Data":"656511351c30c9e795c45e62ed071c0c74153ee895243cd692956100c993b51e"}
Oct 31 16:48:50.843596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:50.843556    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" event={"ID":"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624","Type":"ContainerStarted","Data":"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"}
Oct 31 16:48:50.844107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:50.843729    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:48:50.949869 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:48:50.949814    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" podStartSLOduration=1.691456482 podStartE2EDuration="3.949801589s" totalImagesPullingTime="2.258345107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:48:50.948490085 +0000 UTC m=+2776.010065272" watchObservedRunningTime="2026-10-31 16:48:50.949801589 +0000 UTC m=+2776.011376775"
Oct 31 16:49:01.850951 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:01.850915    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:49:02.942279 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:49:02.942243    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:49:03.036498 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.036467    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"]
Oct 31 16:49:03.041254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.041231    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.089716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.089681    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 16:49:03.089716 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.089684    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gw56j\""
Oct 31 16:49:03.125793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.125768    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.125956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.125813    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21201683-d4a2-4c03-a84f-ff1803240663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.125956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.125833    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21201683-d4a2-4c03-a84f-ff1803240663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.125956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.125916    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.126069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.125985    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.126069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.126030    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.126136 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.126120    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7bg\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-kube-api-access-kj7bg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.126174 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.126151    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.126209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.126172    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.144325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.144289    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"]
Oct 31 16:49:03.227488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227403    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7bg\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-kube-api-access-kj7bg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227447    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227488 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227476    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227503    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227549    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21201683-d4a2-4c03-a84f-ff1803240663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227664    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21201683-d4a2-4c03-a84f-ff1803240663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227720    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227748    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.227896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227789    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.228191 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.227966    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.228243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.228212    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.228243 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.228229    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.228424 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.228401    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21201683-d4a2-4c03-a84f-ff1803240663-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.228466 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.228414    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.229929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.229904    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21201683-d4a2-4c03-a84f-ff1803240663-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.230201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.230181    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21201683-d4a2-4c03-a84f-ff1803240663-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.280252 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.280219    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.280457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.280430    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7bg\" (UniqueName: \"kubernetes.io/projected/21201683-d4a2-4c03-a84f-ff1803240663-kube-api-access-kj7bg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8965v\" (UID: \"21201683-d4a2-4c03-a84f-ff1803240663\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.355322 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.355282    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:03.770802 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.770764    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"]
Oct 31 16:49:03.778319 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:49:03.778292    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21201683_d4a2_4c03_a84f_ff1803240663.slice/crio-1a91c0e39f5edca7189b6fef41207cba4c74382c367423042392b3fa19a004bb WatchSource:0}: Error finding container 1a91c0e39f5edca7189b6fef41207cba4c74382c367423042392b3fa19a004bb: Status 404 returned error can't find the container with id 1a91c0e39f5edca7189b6fef41207cba4c74382c367423042392b3fa19a004bb
Oct 31 16:49:03.780468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.780432    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:03.780552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.780499    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:03.780552 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.780529    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:03.898947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.898908    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" event={"ID":"21201683-d4a2-4c03-a84f-ff1803240663","Type":"ContainerStarted","Data":"426e2f158d2bda9110c6e9b6bbf72a7bb17aa6e7915f6d46b9aeaa5693dc1f54"}
Oct 31 16:49:03.899122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:03.898971    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" event={"ID":"21201683-d4a2-4c03-a84f-ff1803240663","Type":"ContainerStarted","Data":"1a91c0e39f5edca7189b6fef41207cba4c74382c367423042392b3fa19a004bb"}
Oct 31 16:49:04.006296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:04.006246    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podStartSLOduration=2.006232546 podStartE2EDuration="2.006232546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:49:04.004859961 +0000 UTC m=+2789.066435142" watchObservedRunningTime="2026-10-31 16:49:04.006232546 +0000 UTC m=+2789.067807734"
Oct 31 16:49:04.355443 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:04.355400    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:04.357305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:04.357280    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:04.357433 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:04.357321    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:05.356733 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:05.356686    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:05.357327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:05.357299    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:06.357097 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:06.357064    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:06.357727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:06.357696    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:07.358072 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:07.358033    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:07.358701 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:07.358664    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:08.356639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:08.355980    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:08.356639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:08.356045    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:09.358640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:09.357995    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:09.358640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:09.358053    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:10.356572 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.356532    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:10.356780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.356641    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:10.434685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.434648    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"]
Oct 31 16:49:10.438680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.438653    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.486779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.486748    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:49:10.502583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.502557    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:49:10.502722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.502639    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:49:10.509132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.509107    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.509308 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.509238    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6w89\" (UniqueName: \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.509389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.509325    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.532746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.532723    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:49:10.534575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.534552    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"]
Oct 31 16:49:10.610381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.610296    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6w89\" (UniqueName: \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.610381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.610353    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.610381 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.610379    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.611300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.611273    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.612683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.612661    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.662847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.662814    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6w89\" (UniqueName: \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\") pod \"maas-api-revoke-sub-85237e45-s46gd\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:10.752459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:10.752423    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:11.123108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:11.123070    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"]
Oct 31 16:49:11.130467 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:49:11.130437    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167f9398_1566_4843_9004_15bc67006643.slice/crio-f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033 WatchSource:0}: Error finding container f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033: Status 404 returned error can't find the container with id f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033
Oct 31 16:49:11.356528 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:11.356502    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 16:49:11.356671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:11.356553    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 16:49:11.933277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:11.933240    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd" event={"ID":"167f9398-1566-4843-9004-15bc67006643","Type":"ContainerStarted","Data":"a3a1ecd732ac9affd7f093a03dfe45a98d40e09a109a915c4aba54567d289f88"}
Oct 31 16:49:11.933862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:11.933298    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd" event={"ID":"167f9398-1566-4843-9004-15bc67006643","Type":"ContainerStarted","Data":"f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033"}
Oct 31 16:49:12.037562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.037513    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd" podStartSLOduration=2.037501675 podStartE2EDuration="2.037501675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:49:12.036520646 +0000 UTC m=+2797.098095830" watchObservedRunningTime="2026-10-31 16:49:12.037501675 +0000 UTC m=+2797.099076897"
Oct 31 16:49:12.359065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.359036    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:12.359330 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.359306    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:12.360019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.360002    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v"
Oct 31 16:49:12.938550 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.938516    4417 generic.go:356] "Generic (PLEG): container finished" podID="167f9398-1566-4843-9004-15bc67006643" containerID="a3a1ecd732ac9affd7f093a03dfe45a98d40e09a109a915c4aba54567d289f88" exitCode=0
Oct 31 16:49:12.938989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:12.938600    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd" event={"ID":"167f9398-1566-4843-9004-15bc67006643","Type":"ContainerDied","Data":"a3a1ecd732ac9affd7f093a03dfe45a98d40e09a109a915c4aba54567d289f88"}
Oct 31 16:49:14.090988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.090962    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:14.141113 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.141088    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\") pod \"167f9398-1566-4843-9004-15bc67006643\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") "
Oct 31 16:49:14.141275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.141147    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\") pod \"167f9398-1566-4843-9004-15bc67006643\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") "
Oct 31 16:49:14.141275 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.141234    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\" (UniqueName: \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\") pod \"167f9398-1566-4843-9004-15bc67006643\" (UID: \"167f9398-1566-4843-9004-15bc67006643\") "
Oct 31 16:49:14.141432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.141408    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca" pod "167f9398-1566-4843-9004-15bc67006643" (UID: "167f9398-1566-4843-9004-15bc67006643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:49:14.141514 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.141500    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/167f9398-1566-4843-9004-15bc67006643-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:14.143247 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.143217    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca" pod "167f9398-1566-4843-9004-15bc67006643" (UID: "167f9398-1566-4843-9004-15bc67006643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:49:14.143358 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.143304    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89" pod "167f9398-1566-4843-9004-15bc67006643" (UID: "167f9398-1566-4843-9004-15bc67006643"). InnerVolumeSpecName "kube-api-access-b6w89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:49:14.242744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.242666    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/167f9398-1566-4843-9004-15bc67006643-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:14.242744 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.242697    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6w89\" (UniqueName: \"kubernetes.io/projected/167f9398-1566-4843-9004-15bc67006643-kube-api-access-b6w89\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:14.948562 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.948532    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"
Oct 31 16:49:14.948734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.948531    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd" event={"ID":"167f9398-1566-4843-9004-15bc67006643","Type":"ContainerDied","Data":"f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033"}
Oct 31 16:49:14.948734 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:14.948702    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9db3e56d2c41dd43f7c37c25a358789f7764588023036037b39300f8458b033"
Oct 31 16:49:18.151049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.151014    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"]
Oct 31 16:49:18.151479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.151366    4417 cpu_manager.go:414] "containerMap: removing container" podUID="167f9398-1566-4843-9004-15bc67006643" containerName="revoke-keys"
Oct 31 16:49:18.151479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.151377    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167f9398-1566-4843-9004-15bc67006643" containerName="revoke-keys"
Oct 31 16:49:18.151479 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.151445    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="167f9398-1566-4843-9004-15bc67006643" containerName="revoke-keys"
Oct 31 16:49:18.154705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.154679    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.177687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177664    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.177814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177694    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbkd\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-kube-api-access-7pbkd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.177814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177716    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.177917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177819    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.178012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177923    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec81237f-b4d2-4591-8e0d-43c7a20df439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.178012 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.177970    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.178105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.178008    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.178105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.178042    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.178188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.178169    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.250224 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.250193    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"]
Oct 31 16:49:18.278742 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278709    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec81237f-b4d2-4591-8e0d-43c7a20df439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.278903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278752    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.278903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278778    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.278903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278801    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.278903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278830    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.278903 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.278859    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279074    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbkd\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-kube-api-access-7pbkd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279250    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279263    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279375    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279430 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279383    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279457    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279484    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec81237f-b4d2-4591-8e0d-43c7a20df439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.279774 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.279755    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.281822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.281798    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.281963 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.281946    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.331465 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.331431    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.331583 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.331518    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbkd\" (UniqueName: \"kubernetes.io/projected/ec81237f-b4d2-4591-8e0d-43c7a20df439-kube-api-access-7pbkd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-dx8zr\" (UID: \"ec81237f-b4d2-4591-8e0d-43c7a20df439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.467718 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.467643    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:18.883510 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.883476    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"]
Oct 31 16:49:18.890804 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:49:18.890777    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec81237f_b4d2_4591_8e0d_43c7a20df439.slice/crio-d3889ac77294e753a54a73e3788939712441eca127b6f04e2ae3070b27943004 WatchSource:0}: Error finding container d3889ac77294e753a54a73e3788939712441eca127b6f04e2ae3070b27943004: Status 404 returned error can't find the container with id d3889ac77294e753a54a73e3788939712441eca127b6f04e2ae3070b27943004
Oct 31 16:49:18.892846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.892814    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:18.892948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.892887    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:18.892948 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.892918    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:49:18.969040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.969006    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" event={"ID":"ec81237f-b4d2-4591-8e0d-43c7a20df439","Type":"ContainerStarted","Data":"3c94b4db69dbf91ee71ee04caf05976b68b1e3d26f495f5c4e943f07c26da444"}
Oct 31 16:49:18.969181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:18.969059    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" event={"ID":"ec81237f-b4d2-4591-8e0d-43c7a20df439","Type":"ContainerStarted","Data":"d3889ac77294e753a54a73e3788939712441eca127b6f04e2ae3070b27943004"}
Oct 31 16:49:19.077822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:19.077749    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podStartSLOduration=1.077729337 podStartE2EDuration="1.077729337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:49:19.075702284 +0000 UTC m=+2804.137277469" watchObservedRunningTime="2026-10-31 16:49:19.077729337 +0000 UTC m=+2804.139304521"
Oct 31 16:49:19.468058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:19.468016    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:19.469579 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:19.469557    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:19.469665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:19.469604    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:20.468856 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:20.468821    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:20.469340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:20.468878    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:21.468537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:21.468498    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:21.468739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:21.468556    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:22.468683 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:22.468652    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:22.469084 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:22.468706    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:23.075825 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.075790    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:49:23.076005 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.075977    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" podUID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" containerName="manager" containerID="cri-o://74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68" gracePeriod=30
Oct 31 16:49:23.452639 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.450369    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:49:23.468547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.468521    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:23.468694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.468569    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:23.527185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.527158    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\" (UniqueName: \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\") pod \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") "
Oct 31 16:49:23.527499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.527206    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\") pod \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") "
Oct 31 16:49:23.527499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.527238    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\") pod \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\" (UID: \"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624\") "
Oct 31 16:49:23.529231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.529201    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs" pod "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" (UID: "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:49:23.529339 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.529248    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert" pod "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" (UID: "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:49:23.529392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.529366    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp" pod "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" (UID: "f7e4da77-beeb-4fd0-ac02-8aa4c7a58624"). InnerVolumeSpecName "kube-api-access-h5djp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:49:23.628642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.628555    4417 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-webhook-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:23.628642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.628581    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:23.628642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.628591    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5djp\" (UniqueName: \"kubernetes.io/projected/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624-kube-api-access-h5djp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:49:23.990527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.990439    4417 generic.go:356] "Generic (PLEG): container finished" podID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" containerID="74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68" exitCode=0
Oct 31 16:49:23.990527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.990478    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-ttc9p"
Oct 31 16:49:23.990527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.990514    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" event={"ID":"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624","Type":"ContainerDied","Data":"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"}
Oct 31 16:49:23.990829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.990567    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-ttc9p" event={"ID":"f7e4da77-beeb-4fd0-ac02-8aa4c7a58624","Type":"ContainerDied","Data":"656511351c30c9e795c45e62ed071c0c74153ee895243cd692956100c993b51e"}
Oct 31 16:49:23.990829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.990590    4417 scope.go:122] "RemoveContainer" containerID="74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"
Oct 31 16:49:23.999793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:23.999775    4417 scope.go:122] "RemoveContainer" containerID="74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"
Oct 31 16:49:24.000061 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:49:24.000045    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68\": container with ID starting with 74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68 not found: ID does not exist" containerID="74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"
Oct 31 16:49:24.000135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:24.000068    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68"} err="failed to get container status \"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68\": rpc error: code = NotFound desc = could not find container \"74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68\": container with ID starting with 74bd9f995a957eff1fbfe4691327a85feae660bc17bad1457ee216f4aaee0e68 not found: ID does not exist"
Oct 31 16:49:24.157640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:24.157388    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:49:24.202729 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:24.202698    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-ttc9p"]
Oct 31 16:49:24.468148 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:24.468115    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:24.468342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:24.468166    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:25.052968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.052933    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" path="/var/lib/kubelet/pods/f7e4da77-beeb-4fd0-ac02-8aa4c7a58624/volumes"
Oct 31 16:49:25.468258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.468181    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 16:49:25.468258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.468234    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 16:49:25.545317 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.545291    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c95c64d65-mkxsh"]
Oct 31 16:49:25.545664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.545651    4417 cpu_manager.go:414] "containerMap: removing container" podUID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" containerName="manager"
Oct 31 16:49:25.545714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.545665    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" containerName="manager"
Oct 31 16:49:25.545765 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.545724    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e4da77-beeb-4fd0-ac02-8aa4c7a58624" containerName="manager"
Oct 31 16:49:25.551067 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.551047    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.599505 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.599482    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 16:49:25.614411 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.614388    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v4hcf\""
Oct 31 16:49:25.645477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.645453    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 16:49:25.645477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.645457    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 16:49:25.649754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.649730    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-mkxsh"]
Oct 31 16:49:25.671737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.671712    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcdb\" (UniqueName: \"kubernetes.io/projected/d37a89b0-513a-4eff-be99-6758193b06b0-kube-api-access-skcdb\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.671845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.671753    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-metrics-certs\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.671894 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.671833    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-webhook-cert\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.772851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.772775    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-metrics-certs\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.772851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.772822    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-webhook-cert\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.773043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.772940    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcdb\" (UniqueName: \"kubernetes.io/projected/d37a89b0-513a-4eff-be99-6758193b06b0-kube-api-access-skcdb\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.775349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.775324    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-metrics-certs\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.775444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.775374    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d37a89b0-513a-4eff-be99-6758193b06b0-webhook-cert\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.825532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.825502    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcdb\" (UniqueName: \"kubernetes.io/projected/d37a89b0-513a-4eff-be99-6758193b06b0-kube-api-access-skcdb\") pod \"maas-controller-6c95c64d65-mkxsh\" (UID: \"d37a89b0-513a-4eff-be99-6758193b06b0\") " pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:25.864201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:25.864169    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:26.234244 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:26.234201    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-mkxsh"]
Oct 31 16:49:26.241926 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:49:26.241898    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37a89b0_513a_4eff_be99_6758193b06b0.slice/crio-5422c6a78f0acc057b26aced8db30c747086508c9e8a8db2e6d8ef46c7bd5956 WatchSource:0}: Error finding container 5422c6a78f0acc057b26aced8db30c747086508c9e8a8db2e6d8ef46c7bd5956: Status 404 returned error can't find the container with id 5422c6a78f0acc057b26aced8db30c747086508c9e8a8db2e6d8ef46c7bd5956
Oct 31 16:49:26.955349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:26.955317    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:26.955674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:26.955650    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:26.956594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:26.956574    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr"
Oct 31 16:49:27.006133 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:27.006106    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-mkxsh" event={"ID":"d37a89b0-513a-4eff-be99-6758193b06b0","Type":"ContainerStarted","Data":"5422c6a78f0acc057b26aced8db30c747086508c9e8a8db2e6d8ef46c7bd5956"}
Oct 31 16:49:28.012003 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:28.011970    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-mkxsh" event={"ID":"d37a89b0-513a-4eff-be99-6758193b06b0","Type":"ContainerStarted","Data":"d6a469387a753e78471d28fb9ba84ecde9bb71eb6b724846cb5a18aa95283ca4"}
Oct 31 16:49:28.012601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:28.012053    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:49:28.119813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:28.119764    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c95c64d65-mkxsh" podStartSLOduration=2.245239112 podStartE2EDuration="3.119751285s" totalImagesPullingTime="874.512173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:49:28.118458534 +0000 UTC m=+2813.180033720" watchObservedRunningTime="2026-10-31 16:49:28.119751285 +0000 UTC m=+2813.181326468"
Oct 31 16:49:32.944349 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:49:32.944317    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:49:39.018110 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:49:39.018077    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c95c64d65-mkxsh"
Oct 31 16:50:02.944166 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:50:02.944092    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:50:13.400837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:13.400792    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:13.401379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:13.400867    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:19.555947 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.555909    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"]
Oct 31 16:50:19.559386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.559363    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.653463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.653400    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:50:19.653463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.653418    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:50:19.653463 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.653461    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:50:19.653823 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.653467    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:50:19.655808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.655784    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"]
Oct 31 16:50:19.665739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.665704    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.665883 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.665760    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.665929 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.665889    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xtb\" (UniqueName: \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.766820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.766777    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.767008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.766828    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.767008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.766876    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xtb\" (UniqueName: \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.767580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.767559    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.769509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.769488    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.819374 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.819298    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xtb\" (UniqueName: \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\") pod \"maas-api-revoke-sub-c1e271a9-hgntc\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:19.872221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:19.872186    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:20.235972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:20.235869    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"]
Oct 31 16:50:20.243710 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:50:20.243682    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0accfb4_e930_4fd8_a265_fa1762ca794f.slice/crio-7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957 WatchSource:0}: Error finding container 7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957: Status 404 returned error can't find the container with id 7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957
Oct 31 16:50:21.246162 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:21.246129    4417 generic.go:356] "Generic (PLEG): container finished" podID="e0accfb4-e930-4fd8-a265-fa1762ca794f" containerID="dbbc858318a088836b670a4491ea6e553dfd6480cc39e38cafb427283e017fce" exitCode=0
Oct 31 16:50:21.246601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:21.246218    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc" event={"ID":"e0accfb4-e930-4fd8-a265-fa1762ca794f","Type":"ContainerDied","Data":"dbbc858318a088836b670a4491ea6e553dfd6480cc39e38cafb427283e017fce"}
Oct 31 16:50:21.246601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:21.246289    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc" event={"ID":"e0accfb4-e930-4fd8-a265-fa1762ca794f","Type":"ContainerStarted","Data":"7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957"}
Oct 31 16:50:22.380098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.380073    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:22.494820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.494772    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\") pod \"e0accfb4-e930-4fd8-a265-fa1762ca794f\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") "
Oct 31 16:50:22.495022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.494846    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\" (UniqueName: \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\") pod \"e0accfb4-e930-4fd8-a265-fa1762ca794f\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") "
Oct 31 16:50:22.495022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.494930    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\") pod \"e0accfb4-e930-4fd8-a265-fa1762ca794f\" (UID: \"e0accfb4-e930-4fd8-a265-fa1762ca794f\") "
Oct 31 16:50:22.495213 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.495190    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca" pod "e0accfb4-e930-4fd8-a265-fa1762ca794f" (UID: "e0accfb4-e930-4fd8-a265-fa1762ca794f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:50:22.497183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.497154    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb" pod "e0accfb4-e930-4fd8-a265-fa1762ca794f" (UID: "e0accfb4-e930-4fd8-a265-fa1762ca794f"). InnerVolumeSpecName "kube-api-access-b7xtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:50:22.497331 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.497279    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca" pod "e0accfb4-e930-4fd8-a265-fa1762ca794f" (UID: "e0accfb4-e930-4fd8-a265-fa1762ca794f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:50:22.595813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.595769    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xtb\" (UniqueName: \"kubernetes.io/projected/e0accfb4-e930-4fd8-a265-fa1762ca794f-kube-api-access-b7xtb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:22.595813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.595807    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:22.595813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:22.595818    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0accfb4-e930-4fd8-a265-fa1762ca794f-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:23.256277 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:23.256248    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"
Oct 31 16:50:23.256469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:23.256247    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc" event={"ID":"e0accfb4-e930-4fd8-a265-fa1762ca794f","Type":"ContainerDied","Data":"7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957"}
Oct 31 16:50:23.256469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:23.256375    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7787f21e4ac45febb7cf926d0ea7494d7fdd136d40ba16599a7f56baca142957"
Oct 31 16:50:32.943721 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:50:32.943685    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:50:33.717359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.717325    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"]
Oct 31 16:50:33.717753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.717729    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e0accfb4-e930-4fd8-a265-fa1762ca794f" containerName="revoke-keys"
Oct 31 16:50:33.717809 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.717754    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0accfb4-e930-4fd8-a265-fa1762ca794f" containerName="revoke-keys"
Oct 31 16:50:33.717847 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.717829    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0accfb4-e930-4fd8-a265-fa1762ca794f" containerName="revoke-keys"
Oct 31 16:50:33.720851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.720828    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.763352 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.763311    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"]
Oct 31 16:50:33.766767 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.766748    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789235 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789202    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.789392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789248    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.789392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789277    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.789392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789305    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zm2s\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-kube-api-access-6zm2s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789392 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789385    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgn26\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-kube-api-access-cgn26\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789414    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a05f7-0e52-4001-88c9-b40cc107b30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789440    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789477    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789503    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789535    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789570    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789601    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.789680 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789656    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.790168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789697    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.790168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789730    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.790168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789778    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.790168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789812    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.790168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.789842    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.890323 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890278    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn26\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-kube-api-access-cgn26\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890334    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a05f7-0e52-4001-88c9-b40cc107b30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890365    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890398    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890426    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890455    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890504 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890490    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.890904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890521    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.890904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890547    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.890904 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.890587    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891021    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.891085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891075    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.891200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891119    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891138    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891151    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.891200 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891182    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891221    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891231    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a05f7-0e52-4001-88c9-b40cc107b30a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.891437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891252    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891284    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891437 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891314    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zm2s\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-kube-api-access-6zm2s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.891815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891494    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.891815 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.891775    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.892038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.892016    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.892171 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.892074    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.892272 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.892248    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.892494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.892466    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.892586 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.892505    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.893850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.893824    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.894918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.894893    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.895222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.895196    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.895325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.895298    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.913068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.913047    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"]
Oct 31 16:50:33.930655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.930608    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"]
Oct 31 16:50:33.948019 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.947964    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:33.959986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.959936    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn26\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-kube-api-access-cgn26\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.961688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.961644    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba404bf8-c02c-4e49-b4a8-41968ddb20b1-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-x6tl6\" (UID: \"ba404bf8-c02c-4e49-b4a8-41968ddb20b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:33.963496 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:33.963441    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zm2s\" (UniqueName: \"kubernetes.io/projected/312a05f7-0e52-4001-88c9-b40cc107b30a-kube-api-access-6zm2s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-s8bll\" (UID: \"312a05f7-0e52-4001-88c9-b40cc107b30a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:34.033017 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.032946    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:34.083609 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.083578    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:34.469333 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.469297    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"]
Oct 31 16:50:34.516710 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:50:34.516677    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba404bf8_c02c_4e49_b4a8_41968ddb20b1.slice/crio-0a17e73e6770254749b92b29f7234c3d7a844cede1b75348501194a5d1be9d99 WatchSource:0}: Error finding container 0a17e73e6770254749b92b29f7234c3d7a844cede1b75348501194a5d1be9d99: Status 404 returned error can't find the container with id 0a17e73e6770254749b92b29f7234c3d7a844cede1b75348501194a5d1be9d99
Oct 31 16:50:34.518764 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.518731    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.518884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.518813    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.518884 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.518856    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.523121 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.523099    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"]
Oct 31 16:50:34.529842 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:50:34.529710    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312a05f7_0e52_4001_88c9_b40cc107b30a.slice/crio-0c732370a8ffb4a1757fd656ae123fff89e0a295be2336ef93449ce204f17263 WatchSource:0}: Error finding container 0c732370a8ffb4a1757fd656ae123fff89e0a295be2336ef93449ce204f17263: Status 404 returned error can't find the container with id 0c732370a8ffb4a1757fd656ae123fff89e0a295be2336ef93449ce204f17263
Oct 31 16:50:34.531756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.531728    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.531852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.531799    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:34.531852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:34.531842    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:50:35.308509 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.308467    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" event={"ID":"312a05f7-0e52-4001-88c9-b40cc107b30a","Type":"ContainerStarted","Data":"32761d96418db8fe57a5d869473413b0d1af0ded59f63417e18f2eb83d6f8400"}
Oct 31 16:50:35.308934 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.308526    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" event={"ID":"312a05f7-0e52-4001-88c9-b40cc107b30a","Type":"ContainerStarted","Data":"0c732370a8ffb4a1757fd656ae123fff89e0a295be2336ef93449ce204f17263"}
Oct 31 16:50:35.310750 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.310728    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" event={"ID":"ba404bf8-c02c-4e49-b4a8-41968ddb20b1","Type":"ContainerStarted","Data":"70ca022127d276f2cec44827592a51b9f7c70160d6316535409a8b3751ad8d30"}
Oct 31 16:50:35.310807 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.310768    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" event={"ID":"ba404bf8-c02c-4e49-b4a8-41968ddb20b1","Type":"ContainerStarted","Data":"0a17e73e6770254749b92b29f7234c3d7a844cede1b75348501194a5d1be9d99"}
Oct 31 16:50:35.511061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.511009    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podStartSLOduration=2.510993739 podStartE2EDuration="2.510993739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:35.464178487 +0000 UTC m=+2880.525753693" watchObservedRunningTime="2026-10-31 16:50:35.510993739 +0000 UTC m=+2880.572568924"
Oct 31 16:50:35.571231 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:35.571114    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podStartSLOduration=2.571094098 podStartE2EDuration="2.571094098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:35.570348557 +0000 UTC m=+2880.631923740" watchObservedRunningTime="2026-10-31 16:50:35.571094098 +0000 UTC m=+2880.632669286"
Oct 31 16:50:36.033489 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.033396    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:36.055835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.055779    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:36.056167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.056126    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:36.084922 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.084884    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:36.086917 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.086886    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:36.087142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:36.087108    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:37.037644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:37.033902    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:37.037644 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:37.033959    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:37.084916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:37.084291    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:37.084916 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:37.084356    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:38.043642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:38.037940    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:38.043642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:38.038000    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:38.087053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:38.087023    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:38.087273 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:38.087253    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:39.033873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:39.033839    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:39.034176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:39.034110    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:39.085210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:39.085177    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:39.086258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:39.085764    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:40.037741 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:40.037696    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:40.038049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:40.038017    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:40.092386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:40.086064    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:40.092386 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:40.086173    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:41.033524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:41.033489    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:41.033731 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:41.033566    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:41.084746 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:41.084715    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:41.084896 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:41.084776    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:42.043779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:42.043740    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 16:50:42.044441 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:42.044389    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 16:50:42.089641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:42.088714    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 16:50:42.089641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:42.088783    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 16:50:43.406646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:43.406137    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:43.406646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:43.406212    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:44.033840 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:44.033801    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:44.034049 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:44.033861    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:44.091418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:44.091373    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:44.091606 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:44.091420    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:45.074849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:45.074755    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:45.074849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:45.074831    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:45.132757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:45.132724    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:45.132940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:45.132773    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:46.115780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.115744    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:46.116384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.116336    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:46.174792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.174751    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:50:46.174992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.174809    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:50:46.251776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.251738    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:46.252170 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.252146    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:46.254897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.254874    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6"
Oct 31 16:50:46.284846 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.284823    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:46.285185 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.285158    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:46.285832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:46.285818    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll"
Oct 31 16:50:50.939156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:50.939124    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"]
Oct 31 16:50:50.942800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:50.942777    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.036932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.036899    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:50:51.037142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.036926    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:50:51.037220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.036952    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:50:51.037220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.037000    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:50:51.038780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.038762    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"]
Oct 31 16:50:51.054384 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.054359    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.054544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.054394    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zl2\" (UniqueName: \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.054544 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.054416    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.155494 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.155452    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.155705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.155520    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zl2\" (UniqueName: \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.155705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.155544    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.156251 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.156227    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.157861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.157842    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.208130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.208069    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zl2\" (UniqueName: \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\") pod \"maas-api-revoke-sub-2e1f5b5f-dbwm5\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.255417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.255392    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:51.622737 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:51.622705    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"]
Oct 31 16:50:51.629265 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:50:51.629235    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af8c940_ffd2_4cce_bf03_608c04a2389d.slice/crio-1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc WatchSource:0}: Error finding container 1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc: Status 404 returned error can't find the container with id 1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc
Oct 31 16:50:52.397456 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:52.397420    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5" event={"ID":"6af8c940-ffd2-4cce-bf03-608c04a2389d","Type":"ContainerStarted","Data":"1f40f1e857effff89e603fa1ee74e9cb888f675e7c9f775665046358e05434fd"}
Oct 31 16:50:52.397858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:52.397477    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5" event={"ID":"6af8c940-ffd2-4cce-bf03-608c04a2389d","Type":"ContainerStarted","Data":"1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc"}
Oct 31 16:50:52.502350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:52.502307    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5" podStartSLOduration=2.502294918 podStartE2EDuration="2.502294918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:52.501231996 +0000 UTC m=+2897.562807181" watchObservedRunningTime="2026-10-31 16:50:52.502294918 +0000 UTC m=+2897.563870103"
Oct 31 16:50:53.403776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:53.403746    4417 generic.go:356] "Generic (PLEG): container finished" podID="6af8c940-ffd2-4cce-bf03-608c04a2389d" containerID="1f40f1e857effff89e603fa1ee74e9cb888f675e7c9f775665046358e05434fd" exitCode=0
Oct 31 16:50:53.404188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:53.403792    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5" event={"ID":"6af8c940-ffd2-4cce-bf03-608c04a2389d","Type":"ContainerDied","Data":"1f40f1e857effff89e603fa1ee74e9cb888f675e7c9f775665046358e05434fd"}
Oct 31 16:50:54.534206 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.534183    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:54.589673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.589647    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\") pod \"6af8c940-ffd2-4cce-bf03-608c04a2389d\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") "
Oct 31 16:50:54.589813 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.589745    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\" (UniqueName: \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\") pod \"6af8c940-ffd2-4cce-bf03-608c04a2389d\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") "
Oct 31 16:50:54.589886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.589828    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\") pod \"6af8c940-ffd2-4cce-bf03-608c04a2389d\" (UID: \"6af8c940-ffd2-4cce-bf03-608c04a2389d\") "
Oct 31 16:50:54.589974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.589949    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca" pod "6af8c940-ffd2-4cce-bf03-608c04a2389d" (UID: "6af8c940-ffd2-4cce-bf03-608c04a2389d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:50:54.590168 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.590150    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:54.591799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.591772    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2" pod "6af8c940-ffd2-4cce-bf03-608c04a2389d" (UID: "6af8c940-ffd2-4cce-bf03-608c04a2389d"). InnerVolumeSpecName "kube-api-access-h2zl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:50:54.591799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.591783    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca" pod "6af8c940-ffd2-4cce-bf03-608c04a2389d" (UID: "6af8c940-ffd2-4cce-bf03-608c04a2389d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:50:54.691356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.691274    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zl2\" (UniqueName: \"kubernetes.io/projected/6af8c940-ffd2-4cce-bf03-608c04a2389d-kube-api-access-h2zl2\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:54.691356 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:54.691309    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af8c940-ffd2-4cce-bf03-608c04a2389d-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:50:55.413461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:55.413379    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"
Oct 31 16:50:55.413461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:55.413394    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5" event={"ID":"6af8c940-ffd2-4cce-bf03-608c04a2389d","Type":"ContainerDied","Data":"1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc"}
Oct 31 16:50:55.413461 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:50:55.413435    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f6bdb8a1d787ce1486c76e297d25dd6d954a52e64a1b98dd2efc5b27b7e90dc"
Oct 31 16:51:02.943511 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:51:02.943472    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:51:03.487749 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.487714    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"]
Oct 31 16:51:03.488249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.488219    4417 cpu_manager.go:414] "containerMap: removing container" podUID="6af8c940-ffd2-4cce-bf03-608c04a2389d" containerName="revoke-keys"
Oct 31 16:51:03.488249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.488239    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af8c940-ffd2-4cce-bf03-608c04a2389d" containerName="revoke-keys"
Oct 31 16:51:03.488419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.488373    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af8c940-ffd2-4cce-bf03-608c04a2389d" containerName="revoke-keys"
Oct 31 16:51:03.491601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.491573    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.575039 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.574996    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.575212 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.575116    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kd5\" (UniqueName: \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.575267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.575221    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.585970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.585937    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:51:03.585970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.585959    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:51:03.586210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.585976    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:51:03.586210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.585946    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:51:03.587834 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.587814    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"]
Oct 31 16:51:03.675897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.675863    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.676069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.675910    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.676069 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.675954    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kd5\" (UniqueName: \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.676611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.676591    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.678292 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.678273    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.728366 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.728327    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kd5\" (UniqueName: \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\") pod \"maas-api-revoke-sub-ec843018-nv9mj\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:03.806780 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:03.806741    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:04.175295 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:04.175197    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"]
Oct 31 16:51:04.457779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:04.457748    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj" event={"ID":"e076b817-5cd6-450b-bf42-d5bea95ffde0","Type":"ContainerStarted","Data":"ec5bcb3d2fd20099247c8d43143c168b2cae1b73364707c4c5fed376de4bc456"}
Oct 31 16:51:04.457899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:04.457818    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj" event={"ID":"e076b817-5cd6-450b-bf42-d5bea95ffde0","Type":"ContainerStarted","Data":"b407dd292202a5147c2857f826deaca0e35a65b25fdc91a62ada081facbac711"}
Oct 31 16:51:04.563428 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:04.563378    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj" podStartSLOduration=1.5633653189999999 podStartE2EDuration="1.563365319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:51:04.562257755 +0000 UTC m=+2909.623832942" watchObservedRunningTime="2026-10-31 16:51:04.563365319 +0000 UTC m=+2909.624940505"
Oct 31 16:51:05.472473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:05.472437    4417 generic.go:356] "Generic (PLEG): container finished" podID="e076b817-5cd6-450b-bf42-d5bea95ffde0" containerID="ec5bcb3d2fd20099247c8d43143c168b2cae1b73364707c4c5fed376de4bc456" exitCode=0
Oct 31 16:51:05.473176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:05.473151    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj" event={"ID":"e076b817-5cd6-450b-bf42-d5bea95ffde0","Type":"ContainerDied","Data":"ec5bcb3d2fd20099247c8d43143c168b2cae1b73364707c4c5fed376de4bc456"}
Oct 31 16:51:06.633462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.633437    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:06.812338 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.812309    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\" (UniqueName: \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\") pod \"e076b817-5cd6-450b-bf42-d5bea95ffde0\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") "
Oct 31 16:51:06.812520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.812356    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\") pod \"e076b817-5cd6-450b-bf42-d5bea95ffde0\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") "
Oct 31 16:51:06.812520 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.812401    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\") pod \"e076b817-5cd6-450b-bf42-d5bea95ffde0\" (UID: \"e076b817-5cd6-450b-bf42-d5bea95ffde0\") "
Oct 31 16:51:06.812843 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.812813    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca" pod "e076b817-5cd6-450b-bf42-d5bea95ffde0" (UID: "e076b817-5cd6-450b-bf42-d5bea95ffde0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:51:06.814423 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.814394    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca" pod "e076b817-5cd6-450b-bf42-d5bea95ffde0" (UID: "e076b817-5cd6-450b-bf42-d5bea95ffde0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:51:06.814519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.814482    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5" pod "e076b817-5cd6-450b-bf42-d5bea95ffde0" (UID: "e076b817-5cd6-450b-bf42-d5bea95ffde0"). InnerVolumeSpecName "kube-api-access-n9kd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:51:06.913095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.913068    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9kd5\" (UniqueName: \"kubernetes.io/projected/e076b817-5cd6-450b-bf42-d5bea95ffde0-kube-api-access-n9kd5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:06.913095 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.913098    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:06.913290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:06.913113    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e076b817-5cd6-450b-bf42-d5bea95ffde0-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:07.485533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:07.485464    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"
Oct 31 16:51:07.485533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:07.485489    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj" event={"ID":"e076b817-5cd6-450b-bf42-d5bea95ffde0","Type":"ContainerDied","Data":"b407dd292202a5147c2857f826deaca0e35a65b25fdc91a62ada081facbac711"}
Oct 31 16:51:07.485776 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:07.485546    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b407dd292202a5147c2857f826deaca0e35a65b25fdc91a62ada081facbac711"
Oct 31 16:51:32.943609 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:51:32.943572    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:51:41.489864 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.489821    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"]
Oct 31 16:51:41.490300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.490209    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e076b817-5cd6-450b-bf42-d5bea95ffde0" containerName="revoke-keys"
Oct 31 16:51:41.490300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.490220    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e076b817-5cd6-450b-bf42-d5bea95ffde0" containerName="revoke-keys"
Oct 31 16:51:41.490300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.490283    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e076b817-5cd6-450b-bf42-d5bea95ffde0" containerName="revoke-keys"
Oct 31 16:51:41.493627 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.493590    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.542131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.542105    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:51:41.542264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.542141    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:51:41.588028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.587997    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:51:41.588289 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.588270    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:51:41.590166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.590147    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"]
Oct 31 16:51:41.599357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.599339    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.599445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.599371    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.599445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.599400    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lq6f\" (UniqueName: \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.700607 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.700579    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lq6f\" (UniqueName: \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.700760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.700709    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.700760 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.700752    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.701518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.701494    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.703142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.703122    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.753079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.753024    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lq6f\" (UniqueName: \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\") pod \"maas-api-revoke-sub-046e1343-6bhbg\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:41.806899 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:41.806878    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:42.169266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:42.169238    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"]
Oct 31 16:51:42.176651 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:51:42.176566    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06831a9_b894_47c2_b592_40a68d8e5aa6.slice/crio-692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1 WatchSource:0}: Error finding container 692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1: Status 404 returned error can't find the container with id 692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1
Oct 31 16:51:42.652300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:42.652260    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg" event={"ID":"e06831a9-b894-47c2-b592-40a68d8e5aa6","Type":"ContainerStarted","Data":"b527c9cbe5518a2ac08c023ffae6c7d675ecd32c2b32e8f21cc782dcf42dd096"}
Oct 31 16:51:42.652707 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:42.652318    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg" event={"ID":"e06831a9-b894-47c2-b592-40a68d8e5aa6","Type":"ContainerStarted","Data":"692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1"}
Oct 31 16:51:42.758577 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:42.758478    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg" podStartSLOduration=1.7584657080000001 podStartE2EDuration="1.758465708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:51:42.756718643 +0000 UTC m=+2947.818293827" watchObservedRunningTime="2026-10-31 16:51:42.758465708 +0000 UTC m=+2947.820040894"
Oct 31 16:51:43.658663 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:43.658629    4417 generic.go:356] "Generic (PLEG): container finished" podID="e06831a9-b894-47c2-b592-40a68d8e5aa6" containerID="b527c9cbe5518a2ac08c023ffae6c7d675ecd32c2b32e8f21cc782dcf42dd096" exitCode=0
Oct 31 16:51:43.659061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:43.658694    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg" event={"ID":"e06831a9-b894-47c2-b592-40a68d8e5aa6","Type":"ContainerDied","Data":"b527c9cbe5518a2ac08c023ffae6c7d675ecd32c2b32e8f21cc782dcf42dd096"}
Oct 31 16:51:44.790086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.790063    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:44.938560 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.938476    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\") pod \"e06831a9-b894-47c2-b592-40a68d8e5aa6\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") "
Oct 31 16:51:44.938773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.938686    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\" (UniqueName: \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\") pod \"e06831a9-b894-47c2-b592-40a68d8e5aa6\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") "
Oct 31 16:51:44.938773 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.938741    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\") pod \"e06831a9-b894-47c2-b592-40a68d8e5aa6\" (UID: \"e06831a9-b894-47c2-b592-40a68d8e5aa6\") "
Oct 31 16:51:44.939156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.939132    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca" pod "e06831a9-b894-47c2-b592-40a68d8e5aa6" (UID: "e06831a9-b894-47c2-b592-40a68d8e5aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:51:44.940529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.940498    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca" pod "e06831a9-b894-47c2-b592-40a68d8e5aa6" (UID: "e06831a9-b894-47c2-b592-40a68d8e5aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:51:44.940878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:44.940859    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f" pod "e06831a9-b894-47c2-b592-40a68d8e5aa6" (UID: "e06831a9-b894-47c2-b592-40a68d8e5aa6"). InnerVolumeSpecName "kube-api-access-2lq6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:51:45.039941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.039912    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:45.039941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.039939    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06831a9-b894-47c2-b592-40a68d8e5aa6-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:45.039941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.039949    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lq6f\" (UniqueName: \"kubernetes.io/projected/e06831a9-b894-47c2-b592-40a68d8e5aa6-kube-api-access-2lq6f\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:45.671467 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.671439    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"
Oct 31 16:51:45.671690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.671475    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg" event={"ID":"e06831a9-b894-47c2-b592-40a68d8e5aa6","Type":"ContainerDied","Data":"692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1"}
Oct 31 16:51:45.671690 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:45.671518    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692df0bbe674448900e470d3e834cb8e2215edf3f12f656af8dd4fb96b4526b1"
Oct 31 16:51:52.387684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.387647    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"]
Oct 31 16:51:52.388207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.388053    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e06831a9-b894-47c2-b592-40a68d8e5aa6" containerName="revoke-keys"
Oct 31 16:51:52.388207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.388063    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06831a9-b894-47c2-b592-40a68d8e5aa6" containerName="revoke-keys"
Oct 31 16:51:52.388207 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.388145    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06831a9-b894-47c2-b592-40a68d8e5aa6" containerName="revoke-keys"
Oct 31 16:51:52.391412 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.391386    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.410031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.409978    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.410153 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.410137    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.410197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.410168    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlgs\" (UniqueName: \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.441108 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.441081    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:51:52.487020 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.486988    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:51:52.489142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.489118    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"]
Oct 31 16:51:52.511140 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.511120    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.511246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.511154    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlgs\" (UniqueName: \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.511246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.511227    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.513473 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.513454    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.541897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.541869    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:51:52.541992 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.541912    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:51:52.552532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.552510    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.564043 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.564017    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlgs\" (UniqueName: \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\") pod \"maas-api-revoke-sub-d3fba056-b7msz\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:52.704715 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:52.704637    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:53.072726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:53.072686    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"]
Oct 31 16:51:53.080021 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:51:53.079996    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b413afd_cd20_4e60_9f17_288740531e53.slice/crio-8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93 WatchSource:0}: Error finding container 8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93: Status 404 returned error can't find the container with id 8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93
Oct 31 16:51:53.710608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:53.710497    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz" event={"ID":"1b413afd-cd20-4e60-9f17-288740531e53","Type":"ContainerStarted","Data":"68183aa18d25de71b55d6f48d2d526ebe178a64748dd6f7cb59ff038a64ac268"}
Oct 31 16:51:53.710608 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:53.710563    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz" event={"ID":"1b413afd-cd20-4e60-9f17-288740531e53","Type":"ContainerStarted","Data":"8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93"}
Oct 31 16:51:53.815566 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:53.815511    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz" podStartSLOduration=1.815499684 podStartE2EDuration="1.815499684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:51:53.814945271 +0000 UTC m=+2958.876520458" watchObservedRunningTime="2026-10-31 16:51:53.815499684 +0000 UTC m=+2958.877074868"
Oct 31 16:51:54.717722 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:54.717686    4417 generic.go:356] "Generic (PLEG): container finished" podID="1b413afd-cd20-4e60-9f17-288740531e53" containerID="68183aa18d25de71b55d6f48d2d526ebe178a64748dd6f7cb59ff038a64ac268" exitCode=0
Oct 31 16:51:54.718201 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:54.717764    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz" event={"ID":"1b413afd-cd20-4e60-9f17-288740531e53","Type":"ContainerDied","Data":"68183aa18d25de71b55d6f48d2d526ebe178a64748dd6f7cb59ff038a64ac268"}
Oct 31 16:51:56.523654 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.522855    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:51:56.643047 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.642952    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\") pod \"1b413afd-cd20-4e60-9f17-288740531e53\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") "
Oct 31 16:51:56.643641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.643308    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\" (UniqueName: \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\") pod \"1b413afd-cd20-4e60-9f17-288740531e53\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") "
Oct 31 16:51:56.643641 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.643377    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\") pod \"1b413afd-cd20-4e60-9f17-288740531e53\" (UID: \"1b413afd-cd20-4e60-9f17-288740531e53\") "
Oct 31 16:51:56.644238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.644216    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca" pod "1b413afd-cd20-4e60-9f17-288740531e53" (UID: "1b413afd-cd20-4e60-9f17-288740531e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:51:56.668190 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.668144    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs" pod "1b413afd-cd20-4e60-9f17-288740531e53" (UID: "1b413afd-cd20-4e60-9f17-288740531e53"). InnerVolumeSpecName "kube-api-access-fqlgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:51:56.678379 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.678339    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca" pod "1b413afd-cd20-4e60-9f17-288740531e53" (UID: "1b413afd-cd20-4e60-9f17-288740531e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:51:56.767689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.767572    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlgs\" (UniqueName: \"kubernetes.io/projected/1b413afd-cd20-4e60-9f17-288740531e53-kube-api-access-fqlgs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:56.767689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.767608    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:56.767689 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.767644    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b413afd-cd20-4e60-9f17-288740531e53-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:51:56.796675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.792904    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz" event={"ID":"1b413afd-cd20-4e60-9f17-288740531e53","Type":"ContainerDied","Data":"8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93"}
Oct 31 16:51:56.796675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.792988    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a82dec691458504f21ae32c00cf0d3e9a81ff9415c3f953db9d064215184e93"
Oct 31 16:51:56.796675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:51:56.793064    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"
Oct 31 16:52:02.944151 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:52:02.944111    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:52:27.228547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.228516    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"]
Oct 31 16:52:27.228972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.228907    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1b413afd-cd20-4e60-9f17-288740531e53" containerName="revoke-keys"
Oct 31 16:52:27.228972 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.228919    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b413afd-cd20-4e60-9f17-288740531e53" containerName="revoke-keys"
Oct 31 16:52:27.229050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.228982    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b413afd-cd20-4e60-9f17-288740531e53" containerName="revoke-keys"
Oct 31 16:52:27.231835 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.231812    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.326199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.326164    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:52:27.326199 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.326165    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:52:27.326427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.326179    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:52:27.326427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.326213    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:52:27.328204 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.328177    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"]
Oct 31 16:52:27.347534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.347509    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.347671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.347556    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.347671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.347574    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.448597 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.448565    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.448793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.448632    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.448793 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.448650    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.449270 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.449249    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.450901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.450878    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.501063 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.501001    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\") pod \"maas-api-revoke-sub-ae594979-zdwch\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.544756 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.544730    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:27.912469 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.912429    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"]
Oct 31 16:52:27.919536 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:52:27.919508    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b80bf86_c0e9_4b0f_b337_04b65c3b0766.slice/crio-8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5 WatchSource:0}: Error finding container 8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5: Status 404 returned error can't find the container with id 8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5
Oct 31 16:52:27.944928 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:27.944896    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch" event={"ID":"9b80bf86-c0e9-4b0f-b337-04b65c3b0766","Type":"ContainerStarted","Data":"8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5"}
Oct 31 16:52:28.950575 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:28.950500    4417 generic.go:356] "Generic (PLEG): container finished" podID="9b80bf86-c0e9-4b0f-b337-04b65c3b0766" containerID="0e328d0f55826cec0a0e4c2b660e789b34d17c7167b43df09a1e8023bf0a910f" exitCode=0
Oct 31 16:52:28.950950 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:28.950575    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch" event={"ID":"9b80bf86-c0e9-4b0f-b337-04b65c3b0766","Type":"ContainerDied","Data":"0e328d0f55826cec0a0e4c2b660e789b34d17c7167b43df09a1e8023bf0a910f"}
Oct 31 16:52:30.089117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.089090    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:30.175186 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.175159    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\") pod \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") "
Oct 31 16:52:30.175318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.175202    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\") pod \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") "
Oct 31 16:52:30.175318 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.175224    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\") pod \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\" (UID: \"9b80bf86-c0e9-4b0f-b337-04b65c3b0766\") "
Oct 31 16:52:30.175563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.175537    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca" pod "9b80bf86-c0e9-4b0f-b337-04b65c3b0766" (UID: "9b80bf86-c0e9-4b0f-b337-04b65c3b0766"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:52:30.177233 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.177213    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca" pod "9b80bf86-c0e9-4b0f-b337-04b65c3b0766" (UID: "9b80bf86-c0e9-4b0f-b337-04b65c3b0766"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:52:30.177348 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.177331    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm" pod "9b80bf86-c0e9-4b0f-b337-04b65c3b0766" (UID: "9b80bf86-c0e9-4b0f-b337-04b65c3b0766"). InnerVolumeSpecName "kube-api-access-kpdkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:52:30.276388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.276315    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:30.276388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.276340    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:30.276388 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.276352    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/9b80bf86-c0e9-4b0f-b337-04b65c3b0766-kube-api-access-kpdkm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:30.961405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.961365    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch" event={"ID":"9b80bf86-c0e9-4b0f-b337-04b65c3b0766","Type":"ContainerDied","Data":"8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5"}
Oct 31 16:52:30.961405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.961383    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"
Oct 31 16:52:30.961405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:30.961413    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e6cdf2477c0bd6d7fca8dcfd0c9c49db4b5ccf0f74d486c6daa430fe037d1a5"
Oct 31 16:52:32.943575 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:52:32.943537    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:52:47.305694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:47.304715    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:52:47.309636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:47.308972    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:52:47.359529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:47.359495    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:52:47.359845 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:47.359813    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:52:55.408056 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.408014    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"]
Oct 31 16:52:55.408685 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.408665    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9b80bf86-c0e9-4b0f-b337-04b65c3b0766" containerName="revoke-keys"
Oct 31 16:52:55.408745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.408687    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b80bf86-c0e9-4b0f-b337-04b65c3b0766" containerName="revoke-keys"
Oct 31 16:52:55.408812 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.408800    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b80bf86-c0e9-4b0f-b337-04b65c3b0766" containerName="revoke-keys"
Oct 31 16:52:55.412419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.412396    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.460998 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.460968    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:52:55.506913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.506878    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:52:55.506913 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.506877    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:52:55.507139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.506878    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:52:55.509220 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.509199    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"]
Oct 31 16:52:55.557557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.557529    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.557754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.557576    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7p7\" (UniqueName: \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.557754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.557672    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.659209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.659114    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.659209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.659174    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7p7\" (UniqueName: \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.659495 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.659283    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.659831 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.659809    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.661611 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.661592    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.712025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.711984    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7p7\" (UniqueName: \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\") pod \"maas-api-revoke-sub-a4aef7d7-j2rz6\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:55.728669 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:55.728644    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:56.101098 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:56.101061    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"]
Oct 31 16:52:56.111798 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:52:56.111764    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135825d3_60d4_4515_bfe9_ac2139d441da.slice/crio-28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af WatchSource:0}: Error finding container 28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af: Status 404 returned error can't find the container with id 28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af
Oct 31 16:52:56.137865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:56.137835    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6" event={"ID":"135825d3-60d4-4515-bfe9-ac2139d441da","Type":"ContainerStarted","Data":"28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af"}
Oct 31 16:52:57.144045 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:57.144014    4417 generic.go:356] "Generic (PLEG): container finished" podID="135825d3-60d4-4515-bfe9-ac2139d441da" containerID="a536e35604714b032bf9a6e04a8c6cd67a823a3070cd6036f7c9083b06e2e6d9" exitCode=0
Oct 31 16:52:57.144448 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:57.144070    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6" event={"ID":"135825d3-60d4-4515-bfe9-ac2139d441da","Type":"ContainerDied","Data":"a536e35604714b032bf9a6e04a8c6cd67a823a3070cd6036f7c9083b06e2e6d9"}
Oct 31 16:52:58.271988 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.271962    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:52:58.388276 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.388248    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\") pod \"135825d3-60d4-4515-bfe9-ac2139d441da\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") "
Oct 31 16:52:58.388460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.388342    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\") pod \"135825d3-60d4-4515-bfe9-ac2139d441da\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") "
Oct 31 16:52:58.388460 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.388373    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\" (UniqueName: \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\") pod \"135825d3-60d4-4515-bfe9-ac2139d441da\" (UID: \"135825d3-60d4-4515-bfe9-ac2139d441da\") "
Oct 31 16:52:58.388709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.388680    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca" pod "135825d3-60d4-4515-bfe9-ac2139d441da" (UID: "135825d3-60d4-4515-bfe9-ac2139d441da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:52:58.390541 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.390517    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7" pod "135825d3-60d4-4515-bfe9-ac2139d441da" (UID: "135825d3-60d4-4515-bfe9-ac2139d441da"). InnerVolumeSpecName "kube-api-access-tg7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:52:58.390736 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.390716    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca" pod "135825d3-60d4-4515-bfe9-ac2139d441da" (UID: "135825d3-60d4-4515-bfe9-ac2139d441da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:52:58.489304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.489239    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:58.489304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.489264    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7p7\" (UniqueName: \"kubernetes.io/projected/135825d3-60d4-4515-bfe9-ac2139d441da-kube-api-access-tg7p7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:58.489304 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:58.489275    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135825d3-60d4-4515-bfe9-ac2139d441da-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:52:59.154449 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:59.154406    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6" event={"ID":"135825d3-60d4-4515-bfe9-ac2139d441da","Type":"ContainerDied","Data":"28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af"}
Oct 31 16:52:59.154650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:59.154465    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d197099234bbaa0eb4a6b7b799116683ada017579c137da1dd172f460998af"
Oct 31 16:52:59.154650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:52:59.154424    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"
Oct 31 16:53:02.943548 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:53:02.943514    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:53:17.301159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:17.300707    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:53:17.301159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:17.300767    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:53:17.334827 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:17.334790    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:53:17.335086 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:17.335057    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:53:25.048342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:25.048308    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"]
Oct 31 16:53:25.093014 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:25.092979    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcdb602-6mt8g"]
Oct 31 16:53:26.280649 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.280601    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"]
Oct 31 16:53:26.281166 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.281149    4417 cpu_manager.go:414] "containerMap: removing container" podUID="135825d3-60d4-4515-bfe9-ac2139d441da" containerName="revoke-keys"
Oct 31 16:53:26.281253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.281167    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135825d3-60d4-4515-bfe9-ac2139d441da" containerName="revoke-keys"
Oct 31 16:53:26.281327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.281290    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="135825d3-60d4-4515-bfe9-ac2139d441da" containerName="revoke-keys"
Oct 31 16:53:26.284893 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.284870    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.333357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.333329    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:53:26.333545 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.333356    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:53:26.379151 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.379114    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:53:26.379372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.379153    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:53:26.381459 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.381433    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"]
Oct 31 16:53:26.387221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.387194    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.387354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.387308    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.387419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.387399    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28dk\" (UniqueName: \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.488885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.488839    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.489107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.488902    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28dk\" (UniqueName: \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.489107 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.488994    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.489769 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.489742    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.491434 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.491412    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.541985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.541961    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28dk\" (UniqueName: \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\") pod \"maas-api-revoke-sub-79805ca9-tbnt6\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.598052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.598014    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:26.964858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:26.964773    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"]
Oct 31 16:53:26.971702 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:53:26.971675    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d290c53_c596_43b7_999e_7581f51b073d.slice/crio-17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a WatchSource:0}: Error finding container 17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a: Status 404 returned error can't find the container with id 17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a
Oct 31 16:53:27.052556 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:27.052522    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4" path="/var/lib/kubelet/pods/53f8cc5f-bfc7-4c0f-9d98-b4159fc0f4d4/volumes"
Oct 31 16:53:27.323805 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:27.323764    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6" event={"ID":"9d290c53-c596-43b7-999e-7581f51b073d","Type":"ContainerStarted","Data":"45e9fb0d0b245aa1e7ee773c61580030e50d492ecdff0d4c4c02ae5e7be1e074"}
Oct 31 16:53:27.324187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:27.323828    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6" event={"ID":"9d290c53-c596-43b7-999e-7581f51b073d","Type":"ContainerStarted","Data":"17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a"}
Oct 31 16:53:27.436816 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:27.436762    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6" podStartSLOduration=1.436747354 podStartE2EDuration="1.436747354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:53:27.428136112 +0000 UTC m=+3052.489711299" watchObservedRunningTime="2026-10-31 16:53:27.436747354 +0000 UTC m=+3052.498322553"
Oct 31 16:53:28.330274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:28.330243    4417 generic.go:356] "Generic (PLEG): container finished" podID="9d290c53-c596-43b7-999e-7581f51b073d" containerID="45e9fb0d0b245aa1e7ee773c61580030e50d492ecdff0d4c4c02ae5e7be1e074" exitCode=0
Oct 31 16:53:28.330727 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:28.330322    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6" event={"ID":"9d290c53-c596-43b7-999e-7581f51b073d","Type":"ContainerDied","Data":"45e9fb0d0b245aa1e7ee773c61580030e50d492ecdff0d4c4c02ae5e7be1e074"}
Oct 31 16:53:29.462327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.462301    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:29.518343 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.518317    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\") pod \"9d290c53-c596-43b7-999e-7581f51b073d\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") "
Oct 31 16:53:29.518521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.518363    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\" (UniqueName: \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\") pod \"9d290c53-c596-43b7-999e-7581f51b073d\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") "
Oct 31 16:53:29.518521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.518405    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\") pod \"9d290c53-c596-43b7-999e-7581f51b073d\" (UID: \"9d290c53-c596-43b7-999e-7581f51b073d\") "
Oct 31 16:53:29.518841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.518819    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca" pod "9d290c53-c596-43b7-999e-7581f51b073d" (UID: "9d290c53-c596-43b7-999e-7581f51b073d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:53:29.520402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.520366    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk" pod "9d290c53-c596-43b7-999e-7581f51b073d" (UID: "9d290c53-c596-43b7-999e-7581f51b073d"). InnerVolumeSpecName "kube-api-access-d28dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:53:29.520491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.520411    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca" pod "9d290c53-c596-43b7-999e-7581f51b073d" (UID: "9d290c53-c596-43b7-999e-7581f51b073d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:53:29.620088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.620007    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d28dk\" (UniqueName: \"kubernetes.io/projected/9d290c53-c596-43b7-999e-7581f51b073d-kube-api-access-d28dk\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:29.620088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.620040    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:29.620088 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:29.620053    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d290c53-c596-43b7-999e-7581f51b073d-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:30.340008 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:30.339977    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"
Oct 31 16:53:30.340183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:30.340022    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6" event={"ID":"9d290c53-c596-43b7-999e-7581f51b073d","Type":"ContainerDied","Data":"17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a"}
Oct 31 16:53:30.340183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:30.340069    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17eaee233a59764795e9f3b124e0b3df53b96fce41c18db790c62f625a5c1c0a"
Oct 31 16:53:32.944483 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:53:32.944451    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:53:43.408762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:43.408729    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:53:43.409376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:43.409327    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:53:50.152187 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.152151    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"]
Oct 31 16:53:50.152596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.152555    4417 cpu_manager.go:414] "containerMap: removing container" podUID="9d290c53-c596-43b7-999e-7581f51b073d" containerName="revoke-keys"
Oct 31 16:53:50.152596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.152565    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d290c53-c596-43b7-999e-7581f51b073d" containerName="revoke-keys"
Oct 31 16:53:50.152704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.152668    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d290c53-c596-43b7-999e-7581f51b073d" containerName="revoke-keys"
Oct 31 16:53:50.155800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.155777    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.204266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.204238    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:53:50.250279 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.250238    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:53:50.250477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.250321    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:53:50.250477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.250373    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:53:50.252692 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.252673    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"]
Oct 31 16:53:50.259601 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.259578    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.259753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.259638    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.259822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.259764    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5cz\" (UniqueName: \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.360969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.360928    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.360969 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.360976    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5cz\" (UniqueName: \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.361258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.361037    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.361651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.361630    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.363342 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.363312    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.413782 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.413707    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5cz\" (UniqueName: \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\") pod \"maas-api-revoke-sub-0b7e8edb-rbkc4\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.468974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.468937    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:50.834147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:50.834111    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"]
Oct 31 16:53:50.841400 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:53:50.841368    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90bc8eb_7ee2_4711_804a_b3a75a797273.slice/crio-20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b WatchSource:0}: Error finding container 20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b: Status 404 returned error can't find the container with id 20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b
Oct 31 16:53:51.454482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:51.454396    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4" event={"ID":"c90bc8eb-7ee2-4711-804a-b3a75a797273","Type":"ContainerStarted","Data":"1fa21593e45d906e91aa9ba291c804cde81321aee5fec6d20983b2fee6794691"}
Oct 31 16:53:51.454482 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:51.454451    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4" event={"ID":"c90bc8eb-7ee2-4711-804a-b3a75a797273","Type":"ContainerStarted","Data":"20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b"}
Oct 31 16:53:51.559866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:51.559802    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4" podStartSLOduration=1.559784508 podStartE2EDuration="1.559784508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:53:51.558331159 +0000 UTC m=+3076.619906344" watchObservedRunningTime="2026-10-31 16:53:51.559784508 +0000 UTC m=+3076.621359691"
Oct 31 16:53:52.462468 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:52.462427    4417 generic.go:356] "Generic (PLEG): container finished" podID="c90bc8eb-7ee2-4711-804a-b3a75a797273" containerID="1fa21593e45d906e91aa9ba291c804cde81321aee5fec6d20983b2fee6794691" exitCode=0
Oct 31 16:53:52.462927 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:52.462511    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4" event={"ID":"c90bc8eb-7ee2-4711-804a-b3a75a797273","Type":"ContainerDied","Data":"1fa21593e45d906e91aa9ba291c804cde81321aee5fec6d20983b2fee6794691"}
Oct 31 16:53:53.604548 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.604527    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:53.790907 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.790877    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\") pod \"c90bc8eb-7ee2-4711-804a-b3a75a797273\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") "
Oct 31 16:53:53.791085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.790946    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\" (UniqueName: \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\") pod \"c90bc8eb-7ee2-4711-804a-b3a75a797273\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") "
Oct 31 16:53:53.791085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.790975    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\") pod \"c90bc8eb-7ee2-4711-804a-b3a75a797273\" (UID: \"c90bc8eb-7ee2-4711-804a-b3a75a797273\") "
Oct 31 16:53:53.791332 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.791303    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca" pod "c90bc8eb-7ee2-4711-804a-b3a75a797273" (UID: "c90bc8eb-7ee2-4711-804a-b3a75a797273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:53:53.793062 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.793035    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz" pod "c90bc8eb-7ee2-4711-804a-b3a75a797273" (UID: "c90bc8eb-7ee2-4711-804a-b3a75a797273"). InnerVolumeSpecName "kube-api-access-xp5cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:53:53.793126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.793106    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca" pod "c90bc8eb-7ee2-4711-804a-b3a75a797273" (UID: "c90bc8eb-7ee2-4711-804a-b3a75a797273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:53:53.891980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.891943    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5cz\" (UniqueName: \"kubernetes.io/projected/c90bc8eb-7ee2-4711-804a-b3a75a797273-kube-api-access-xp5cz\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:53.891980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.891972    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:53.891980 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:53.891981    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90bc8eb-7ee2-4711-804a-b3a75a797273-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:53:54.473665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:54.473632    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"
Oct 31 16:53:54.473665 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:54.473658    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4" event={"ID":"c90bc8eb-7ee2-4711-804a-b3a75a797273","Type":"ContainerDied","Data":"20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b"}
Oct 31 16:53:54.473873 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:53:54.473705    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20eef01da0d3f606da9dbc191ebd42a06fc7f80259fd6feacc36c89fa36bbe2b"
Oct 31 16:54:01.600134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.600100    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"]
Oct 31 16:54:01.600673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.600470    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c90bc8eb-7ee2-4711-804a-b3a75a797273" containerName="revoke-keys"
Oct 31 16:54:01.600673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.600480    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90bc8eb-7ee2-4711-804a-b3a75a797273" containerName="revoke-keys"
Oct 31 16:54:01.600673 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.600563    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90bc8eb-7ee2-4711-804a-b3a75a797273" containerName="revoke-keys"
Oct 31 16:54:01.604023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.603997    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.652557 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.652532    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:54:01.665359 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.665327    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.665519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.665387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.665519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.665418    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbp7l\" (UniqueName: \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.698849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.698820    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:54:01.698849 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.698820    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:54:01.699048 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.698933    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:54:01.700985 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.700966    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"]
Oct 31 16:54:01.766499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.766468    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.766706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.766517    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.766706 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.766640    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbp7l\" (UniqueName: \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.767128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.767107    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.768910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.768886    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.819739 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.819703    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbp7l\" (UniqueName: \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\") pod \"maas-api-revoke-sub-3d4156f1-pcqwz\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:01.918327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:01.918240    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:02.290891 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:02.290864    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"]
Oct 31 16:54:02.298761 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:54:02.298730    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1ea66c_4b10_4880_ad4c_2be2ee38115f.slice/crio-577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77 WatchSource:0}: Error finding container 577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77: Status 404 returned error can't find the container with id 577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77
Oct 31 16:54:02.513071 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:02.513035    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz" event={"ID":"eb1ea66c-4b10-4880-ad4c-2be2ee38115f","Type":"ContainerStarted","Data":"577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77"}
Oct 31 16:54:02.945038 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:54:02.944955    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:54:03.545874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:03.545282    4417 generic.go:356] "Generic (PLEG): container finished" podID="eb1ea66c-4b10-4880-ad4c-2be2ee38115f" containerID="db3774a55b4612f8f73a65241ca7f2e4e7479ba237ea66c96785d505fff85d4b" exitCode=0
Oct 31 16:54:03.545874 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:03.545451    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz" event={"ID":"eb1ea66c-4b10-4880-ad4c-2be2ee38115f","Type":"ContainerDied","Data":"db3774a55b4612f8f73a65241ca7f2e4e7479ba237ea66c96785d505fff85d4b"}
Oct 31 16:54:05.346642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.342818    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:05.490050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.489954    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\") pod \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") "
Oct 31 16:54:05.491117 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.491084    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\" (UniqueName: \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\") pod \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") "
Oct 31 16:54:05.491296 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.491284    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\") pod \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\" (UID: \"eb1ea66c-4b10-4880-ad4c-2be2ee38115f\") "
Oct 31 16:54:05.491974 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.491956    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca" pod "eb1ea66c-4b10-4880-ad4c-2be2ee38115f" (UID: "eb1ea66c-4b10-4880-ad4c-2be2ee38115f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:54:05.504682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.504654    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca" pod "eb1ea66c-4b10-4880-ad4c-2be2ee38115f" (UID: "eb1ea66c-4b10-4880-ad4c-2be2ee38115f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:54:05.504800 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.504777    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l" pod "eb1ea66c-4b10-4880-ad4c-2be2ee38115f" (UID: "eb1ea66c-4b10-4880-ad4c-2be2ee38115f"). InnerVolumeSpecName "kube-api-access-xbp7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:54:05.564753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.564708    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz" event={"ID":"eb1ea66c-4b10-4880-ad4c-2be2ee38115f","Type":"ContainerDied","Data":"577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77"}
Oct 31 16:54:05.564753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.564738    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"
Oct 31 16:54:05.564962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.564778    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="577a0049085ed1b9a070a713a419b3b918c4d528eb03bb3b385ee06a51a31e77"
Oct 31 16:54:05.592516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.592494    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:05.592516 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.592516    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbp7l\" (UniqueName: \"kubernetes.io/projected/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-kube-api-access-xbp7l\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:05.592694 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:05.592526    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1ea66c-4b10-4880-ad4c-2be2ee38115f-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:15.049263 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:15.049225    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"]
Oct 31 16:54:15.089079 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:15.089036    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s46gd"]
Oct 31 16:54:17.062389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:17.061788    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167f9398-1566-4843-9004-15bc67006643" path="/var/lib/kubelet/pods/167f9398-1566-4843-9004-15bc67006643/volumes"
Oct 31 16:54:32.943686 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:54:32.943644    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:54:35.381264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.381222    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"]
Oct 31 16:54:35.426970 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.426937    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pcqwz"]
Oct 31 16:54:35.510476 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.510441    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"]
Oct 31 16:54:35.510865 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.510851    4417 cpu_manager.go:414] "containerMap: removing container" podUID="eb1ea66c-4b10-4880-ad4c-2be2ee38115f" containerName="revoke-keys"
Oct 31 16:54:35.510915 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.510868    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1ea66c-4b10-4880-ad4c-2be2ee38115f" containerName="revoke-keys"
Oct 31 16:54:35.510953 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.510938    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1ea66c-4b10-4880-ad4c-2be2ee38115f" containerName="revoke-keys"
Oct 31 16:54:35.514599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.514577    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.563142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563116    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgbq\" (UniqueName: \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.563334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563157    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.563334 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563288    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.563455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563371    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:54:35.563455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563406    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:54:35.563455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563434    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:54:35.563455 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.563443    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:54:35.571266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.571247    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"]
Oct 31 16:54:35.664407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.664330    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgbq\" (UniqueName: \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.664407 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.664371    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.664651 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.664414    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.665068 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.665047    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.666688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.666669    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.716530 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.716498    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgbq\" (UniqueName: \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\") pod \"maas-api-revoke-sub-3d4156f1-c6xng\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:35.827452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:35.827424    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:36.199910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:36.199880    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"]
Oct 31 16:54:36.206319 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:54:36.206285    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67adc189_7d73_45f2_9be8_64cbe0fac266.slice/crio-0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6 WatchSource:0}: Error finding container 0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6: Status 404 returned error can't find the container with id 0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6
Oct 31 16:54:36.591650 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:36.591606    4417 scope.go:122] "RemoveContainer" containerID="fc6979bcc2127b346511bee0539ffba25e667fe928f85811801eb24dfb8c1a9a"
Oct 31 16:54:36.751112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:36.751033    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng" event={"ID":"67adc189-7d73-45f2-9be8-64cbe0fac266","Type":"ContainerStarted","Data":"2f63c40099e9e828c756825299764dfafbf6648dd2ab077090e34c0831c18393"}
Oct 31 16:54:36.751112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:36.751080    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng" event={"ID":"67adc189-7d73-45f2-9be8-64cbe0fac266","Type":"ContainerStarted","Data":"0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6"}
Oct 31 16:54:36.857493 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:36.857436    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng" podStartSLOduration=1.8574224 podStartE2EDuration="1.8574224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:54:36.855437491 +0000 UTC m=+3121.917012677" watchObservedRunningTime="2026-10-31 16:54:36.8574224 +0000 UTC m=+3121.918997627"
Oct 31 16:54:37.048736 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:37.048701    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1ea66c-4b10-4880-ad4c-2be2ee38115f" path="/var/lib/kubelet/pods/eb1ea66c-4b10-4880-ad4c-2be2ee38115f/volumes"
Oct 31 16:54:37.757753 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:37.757713    4417 generic.go:356] "Generic (PLEG): container finished" podID="67adc189-7d73-45f2-9be8-64cbe0fac266" containerID="2f63c40099e9e828c756825299764dfafbf6648dd2ab077090e34c0831c18393" exitCode=0
Oct 31 16:54:37.758152 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:37.757786    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng" event={"ID":"67adc189-7d73-45f2-9be8-64cbe0fac266","Type":"ContainerDied","Data":"2f63c40099e9e828c756825299764dfafbf6648dd2ab077090e34c0831c18393"}
Oct 31 16:54:38.889130 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.889106    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:38.996682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.996649    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\" (UniqueName: \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\") pod \"67adc189-7d73-45f2-9be8-64cbe0fac266\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") "
Oct 31 16:54:38.996886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.996705    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\") pod \"67adc189-7d73-45f2-9be8-64cbe0fac266\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") "
Oct 31 16:54:38.996886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.996732    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\") pod \"67adc189-7d73-45f2-9be8-64cbe0fac266\" (UID: \"67adc189-7d73-45f2-9be8-64cbe0fac266\") "
Oct 31 16:54:38.997150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.997122    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca" pod "67adc189-7d73-45f2-9be8-64cbe0fac266" (UID: "67adc189-7d73-45f2-9be8-64cbe0fac266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:54:38.999139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.999103    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca" pod "67adc189-7d73-45f2-9be8-64cbe0fac266" (UID: "67adc189-7d73-45f2-9be8-64cbe0fac266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:54:38.999139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:38.999122    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq" pod "67adc189-7d73-45f2-9be8-64cbe0fac266" (UID: "67adc189-7d73-45f2-9be8-64cbe0fac266"). InnerVolumeSpecName "kube-api-access-hqgbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:54:39.098429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.098405    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqgbq\" (UniqueName: \"kubernetes.io/projected/67adc189-7d73-45f2-9be8-64cbe0fac266-kube-api-access-hqgbq\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:39.098429 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.098430    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:39.098570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.098439    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67adc189-7d73-45f2-9be8-64cbe0fac266-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:54:39.768327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.768284    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng" event={"ID":"67adc189-7d73-45f2-9be8-64cbe0fac266","Type":"ContainerDied","Data":"0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6"}
Oct 31 16:54:39.768327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.768307    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"
Oct 31 16:54:39.768327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:54:39.768336    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0411e06704ea00524ece719e9268742b2ca27483a70b7ef95db62e02fdcbf2d6"
Oct 31 16:55:02.943892 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:55:02.943854    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:55:17.302285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:17.302241    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:55:17.303055 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:17.303020    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:55:17.329852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:17.329803    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:55:17.330139 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:17.330095    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:55:22.971444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:22.971404    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"]
Oct 31 16:55:22.971897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:22.971818    4417 cpu_manager.go:414] "containerMap: removing container" podUID="67adc189-7d73-45f2-9be8-64cbe0fac266" containerName="revoke-keys"
Oct 31 16:55:22.971897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:22.971831    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67adc189-7d73-45f2-9be8-64cbe0fac266" containerName="revoke-keys"
Oct 31 16:55:22.971994 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:22.971920    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="67adc189-7d73-45f2-9be8-64cbe0fac266" containerName="revoke-keys"
Oct 31 16:55:22.975416 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:22.975393    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.058503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.058466    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtbv\" (UniqueName: \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.058503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.058506    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.058810 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.058652    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.069377 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.069341    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:55:23.069547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.069414    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:55:23.069547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.069498    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:55:23.069547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.069513    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:55:23.071783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.071767    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"]
Oct 31 16:55:23.073664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.073651    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"]
Oct 31 16:55:23.075257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.075238    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hgntc"]
Oct 31 16:55:23.160004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.159970    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtbv\" (UniqueName: \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.160004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.160009    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.160241 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.160048    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.160757 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.160731    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.162349 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.162330    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.213413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.213375    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtbv\" (UniqueName: \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\") pod \"maas-api-revoke-sub-e2fc9c5b-8bt7t\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.290135 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.290101    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:23.657312 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:23.657231    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"]
Oct 31 16:55:23.664211 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:55:23.664181    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1e9742_9ebd_4dc5_9b9b_0f1e1d58db0c.slice/crio-b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe WatchSource:0}: Error finding container b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe: Status 404 returned error can't find the container with id b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe
Oct 31 16:55:24.105269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:24.105232    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t" event={"ID":"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c","Type":"ContainerStarted","Data":"340e475e460812e1281aabdc57dd9bd5ea2c43f3dbfdd6383d15c3e1827bc6f8"}
Oct 31 16:55:24.105705 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:24.105293    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t" event={"ID":"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c","Type":"ContainerStarted","Data":"b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe"}
Oct 31 16:55:24.212491 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:24.212384    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t" podStartSLOduration=2.212372457 podStartE2EDuration="2.212372457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:55:24.209994555 +0000 UTC m=+3169.271569740" watchObservedRunningTime="2026-10-31 16:55:24.212372457 +0000 UTC m=+3169.273947642"
Oct 31 16:55:25.047997 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:25.047955    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0accfb4-e930-4fd8-a265-fa1762ca794f" path="/var/lib/kubelet/pods/e0accfb4-e930-4fd8-a265-fa1762ca794f/volumes"
Oct 31 16:55:25.111497 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:25.111469    4417 generic.go:356] "Generic (PLEG): container finished" podID="7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" containerID="340e475e460812e1281aabdc57dd9bd5ea2c43f3dbfdd6383d15c3e1827bc6f8" exitCode=0
Oct 31 16:55:25.111941 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:25.111521    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t" event={"ID":"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c","Type":"ContainerDied","Data":"340e475e460812e1281aabdc57dd9bd5ea2c43f3dbfdd6383d15c3e1827bc6f8"}
Oct 31 16:55:26.250093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.250063    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:26.391015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.390929    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\" (UniqueName: \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\") pod \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") "
Oct 31 16:55:26.391015 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.390986    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\") pod \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") "
Oct 31 16:55:26.391265 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.391142    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\") pod \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\" (UID: \"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c\") "
Oct 31 16:55:26.391490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.391464    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca" pod "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" (UID: "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:55:26.393120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.393097    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca" pod "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" (UID: "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:55:26.393290 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.393269    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv" pod "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" (UID: "7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c"). InnerVolumeSpecName "kube-api-access-mhtbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:55:26.492714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.492685    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:26.492714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.492709    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtbv\" (UniqueName: \"kubernetes.io/projected/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-kube-api-access-mhtbv\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:26.492714 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:26.492719    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:27.121754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:27.121726    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"
Oct 31 16:55:27.121754 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:27.121749    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t" event={"ID":"7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c","Type":"ContainerDied","Data":"b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe"}
Oct 31 16:55:27.122042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:27.121804    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b70e5af700a6628e8176d81848a2b1772d9c428f39b870259891183149e365fe"
Oct 31 16:55:32.942998 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:55:32.942964    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:55:33.781371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.781337    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"]
Oct 31 16:55:33.783596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.781739    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" containerName="revoke-keys"
Oct 31 16:55:33.783596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.781751    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" containerName="revoke-keys"
Oct 31 16:55:33.783596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.781831    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" containerName="revoke-keys"
Oct 31 16:55:33.785054 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.785029    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:33.879266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.879232    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:55:33.879439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.879272    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:55:33.879439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.879311    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:55:33.879439 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.879370    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:55:33.881635 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.881595    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"]
Oct 31 16:55:33.959799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.959770    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpbt\" (UniqueName: \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:33.960176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.959844    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:33.960176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:33.959905    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.060747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.060725    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.060914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.060762    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.060914 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.060844    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpbt\" (UniqueName: \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.061445 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.061422    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.063042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.063024    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.113237 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.113213    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpbt\" (UniqueName: \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\") pod \"maas-api-revoke-sub-5a29efb3-m42g8\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.400975 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.400896    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:34.766140 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:34.766061    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"]
Oct 31 16:55:34.773068 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:55:34.773042    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35523db_8577_43af_8c8b_975b929f76d9.slice/crio-c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e WatchSource:0}: Error finding container c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e: Status 404 returned error can't find the container with id c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e
Oct 31 16:55:35.159886 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:35.159851    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8" event={"ID":"d35523db-8577-43af-8c8b-975b929f76d9","Type":"ContainerStarted","Data":"0ce861628a18b2f9f70a932607c1086a16ecb81b20285e8ecf2cc0b6e61dd6ea"}
Oct 31 16:55:35.160266 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:35.159913    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8" event={"ID":"d35523db-8577-43af-8c8b-975b929f76d9","Type":"ContainerStarted","Data":"c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e"}
Oct 31 16:55:35.264911 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:35.264861    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8" podStartSLOduration=2.2648467070000002 podStartE2EDuration="2.264846707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:55:35.263092541 +0000 UTC m=+3180.324667727" watchObservedRunningTime="2026-10-31 16:55:35.264846707 +0000 UTC m=+3180.326421909"
Oct 31 16:55:36.165862 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:36.165827    4417 generic.go:356] "Generic (PLEG): container finished" podID="d35523db-8577-43af-8c8b-975b929f76d9" containerID="0ce861628a18b2f9f70a932607c1086a16ecb81b20285e8ecf2cc0b6e61dd6ea" exitCode=0
Oct 31 16:55:36.166347 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:36.165966    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8" event={"ID":"d35523db-8577-43af-8c8b-975b929f76d9","Type":"ContainerDied","Data":"0ce861628a18b2f9f70a932607c1086a16ecb81b20285e8ecf2cc0b6e61dd6ea"}
Oct 31 16:55:36.612253 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:36.612203    4417 scope.go:122] "RemoveContainer" containerID="a3a1ecd732ac9affd7f093a03dfe45a98d40e09a109a915c4aba54567d289f88"
Oct 31 16:55:38.007234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.006945    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:38.157696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.156604    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\") pod \"d35523db-8577-43af-8c8b-975b929f76d9\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") "
Oct 31 16:55:38.157696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.156781    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\") pod \"d35523db-8577-43af-8c8b-975b929f76d9\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") "
Oct 31 16:55:38.157696 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.156838    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\" (UniqueName: \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\") pod \"d35523db-8577-43af-8c8b-975b929f76d9\" (UID: \"d35523db-8577-43af-8c8b-975b929f76d9\") "
Oct 31 16:55:38.193726 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.182298    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt" pod "d35523db-8577-43af-8c8b-975b929f76d9" (UID: "d35523db-8577-43af-8c8b-975b929f76d9"). InnerVolumeSpecName "kube-api-access-nrpbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:55:38.227713 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.227674    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8" event={"ID":"d35523db-8577-43af-8c8b-975b929f76d9","Type":"ContainerDied","Data":"c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e"}
Oct 31 16:55:38.228004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.227982    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23612d7dc6beeadf9ab19612e5edaa062ae2ae7720c84f0a6f6be280efaf48e"
Oct 31 16:55:38.228205 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.228190    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"
Oct 31 16:55:38.233022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.232983    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca" pod "d35523db-8577-43af-8c8b-975b929f76d9" (UID: "d35523db-8577-43af-8c8b-975b929f76d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:55:38.258801 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.258762    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca" pod "d35523db-8577-43af-8c8b-975b929f76d9" (UID: "d35523db-8577-43af-8c8b-975b929f76d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:55:38.281640 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.259979    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:38.281898 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.281875    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpbt\" (UniqueName: \"kubernetes.io/projected/d35523db-8577-43af-8c8b-975b929f76d9-kube-api-access-nrpbt\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:38.282031 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:38.282017    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35523db-8577-43af-8c8b-975b929f76d9-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:55.049065 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:55.048993    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"]
Oct 31 16:55:55.089510 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:55.089480    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-dbwm5"]
Oct 31 16:55:56.119804 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.119773    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"]
Oct 31 16:55:56.165453 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.165421    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c6xng"]
Oct 31 16:55:56.247946 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.247911    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"]
Oct 31 16:55:56.248299 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.248286    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d35523db-8577-43af-8c8b-975b929f76d9" containerName="revoke-keys"
Oct 31 16:55:56.248357 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.248300    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35523db-8577-43af-8c8b-975b929f76d9" containerName="revoke-keys"
Oct 31 16:55:56.248406 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.248363    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35523db-8577-43af-8c8b-975b929f76d9" containerName="revoke-keys"
Oct 31 16:55:56.251838 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.251818    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.300197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.300173    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:55:56.300197 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.300177    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:55:56.300360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.300224    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:55:56.300360 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.300236    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:55:56.308350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.308326    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"]
Oct 31 16:55:56.373878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.373801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.373878 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.373850    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.374116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.374044    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6c6\" (UniqueName: \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.475011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.474978    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6c6\" (UniqueName: \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.475188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.475042    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.475188 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.475091    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.475836 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.475815    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.477345 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.477325    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.527122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.527090    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6c6\" (UniqueName: \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\") pod \"maas-api-revoke-sub-3d4156f1-zwhfn\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.564977 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.564946    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:56.929093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:56.929061    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"]
Oct 31 16:55:56.935993 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:55:56.935963    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1c4cb_485d_480f_a86e_bee097feabfb.slice/crio-097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e WatchSource:0}: Error finding container 097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e: Status 404 returned error can't find the container with id 097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e
Oct 31 16:55:57.048325 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:57.048296    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67adc189-7d73-45f2-9be8-64cbe0fac266" path="/var/lib/kubelet/pods/67adc189-7d73-45f2-9be8-64cbe0fac266/volumes"
Oct 31 16:55:57.048841 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:57.048825    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af8c940-ffd2-4cce-bf03-608c04a2389d" path="/var/lib/kubelet/pods/6af8c940-ffd2-4cce-bf03-608c04a2389d/volumes"
Oct 31 16:55:57.326150 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:57.326114    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn" event={"ID":"5cf1c4cb-485d-480f-a86e-bee097feabfb","Type":"ContainerStarted","Data":"6243c251b9c1929ebaaa78b98d96d99b15a0be1dbbc1cc3de7b54a466484a7c2"}
Oct 31 16:55:57.326483 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:57.326170    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn" event={"ID":"5cf1c4cb-485d-480f-a86e-bee097feabfb","Type":"ContainerStarted","Data":"097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e"}
Oct 31 16:55:57.429956 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:57.429917    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn" podStartSLOduration=1.429904478 podStartE2EDuration="1.429904478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:55:57.429281766 +0000 UTC m=+3202.490856953" watchObservedRunningTime="2026-10-31 16:55:57.429904478 +0000 UTC m=+3202.491479656"
Oct 31 16:55:58.332534 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:58.332501    4417 generic.go:356] "Generic (PLEG): container finished" podID="5cf1c4cb-485d-480f-a86e-bee097feabfb" containerID="6243c251b9c1929ebaaa78b98d96d99b15a0be1dbbc1cc3de7b54a466484a7c2" exitCode=0
Oct 31 16:55:58.332960 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:58.332590    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn" event={"ID":"5cf1c4cb-485d-480f-a86e-bee097feabfb","Type":"ContainerDied","Data":"6243c251b9c1929ebaaa78b98d96d99b15a0be1dbbc1cc3de7b54a466484a7c2"}
Oct 31 16:55:59.474165 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.474138    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:55:59.502167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.502140    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\") pod \"5cf1c4cb-485d-480f-a86e-bee097feabfb\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") "
Oct 31 16:55:59.502278 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.502220    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\" (UniqueName: \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\") pod \"5cf1c4cb-485d-480f-a86e-bee097feabfb\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") "
Oct 31 16:55:59.502278 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.502247    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\") pod \"5cf1c4cb-485d-480f-a86e-bee097feabfb\" (UID: \"5cf1c4cb-485d-480f-a86e-bee097feabfb\") "
Oct 31 16:55:59.502660 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.502611    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca" pod "5cf1c4cb-485d-480f-a86e-bee097feabfb" (UID: "5cf1c4cb-485d-480f-a86e-bee097feabfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:55:59.504159 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.504136    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca" pod "5cf1c4cb-485d-480f-a86e-bee097feabfb" (UID: "5cf1c4cb-485d-480f-a86e-bee097feabfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:55:59.504254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.504220    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6" pod "5cf1c4cb-485d-480f-a86e-bee097feabfb" (UID: "5cf1c4cb-485d-480f-a86e-bee097feabfb"). InnerVolumeSpecName "kube-api-access-dp6c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:55:59.603768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.603698    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:59.603768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.603725    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp6c6\" (UniqueName: \"kubernetes.io/projected/5cf1c4cb-485d-480f-a86e-bee097feabfb-kube-api-access-dp6c6\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:55:59.603768 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:55:59.603740    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cf1c4cb-485d-480f-a86e-bee097feabfb-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:56:00.343728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:00.343673    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn" event={"ID":"5cf1c4cb-485d-480f-a86e-bee097feabfb","Type":"ContainerDied","Data":"097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e"}
Oct 31 16:56:00.343728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:00.343705    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"
Oct 31 16:56:00.343728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:00.343739    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097193e446fc8b670872f51b89575ba44a70fbd9a99d775f7607e2d23784e56e"
Oct 31 16:56:02.943742 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:56:02.943708    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:56:07.041925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:07.041890    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"]
Oct 31 16:56:07.088053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:07.088021    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-nv9mj"]
Oct 31 16:56:09.048490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:09.048461    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e076b817-5cd6-450b-bf42-d5bea95ffde0" path="/var/lib/kubelet/pods/e076b817-5cd6-450b-bf42-d5bea95ffde0/volumes"
Oct 31 16:56:13.004642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:13.004321    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:56:13.004642 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:13.004419    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:56:28.791906 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.791863    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:56:28.792462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.792437    4417 cpu_manager.go:414] "containerMap: removing container" podUID="5cf1c4cb-485d-480f-a86e-bee097feabfb" containerName="revoke-keys"
Oct 31 16:56:28.792462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.792457    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf1c4cb-485d-480f-a86e-bee097feabfb" containerName="revoke-keys"
Oct 31 16:56:28.792595 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.792572    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf1c4cb-485d-480f-a86e-bee097feabfb" containerName="revoke-keys"
Oct 31 16:56:28.795932 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.795901    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843363 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843329    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843384    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbnp\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843548    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843580    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843595    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.843655 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843681    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.844009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843751    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.844009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843804    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.844009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.843828    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.902202 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.902171    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:56:28.945310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945200    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945247    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945310 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945302    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945350    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbnp\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945395    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945418    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945441    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945476    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.945808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.945540    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.946513 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.946477    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f191a7-openshift-default-dockercfg-z49kc\""
Oct 31 16:56:28.946910 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.946884    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.947181 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.947158    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.948327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.947693    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.948327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.947819    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.948327 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.948264    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.950260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.949942    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.953688 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.953260    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.999157 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.999132    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:28.999861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:28.999837    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbnp\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\") pod \"e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") " pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:29.108829 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:29.108797    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:29.531383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:29.531350    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:56:29.538183 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:56:29.538159    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01757844_146f_4ebe_a426_51f6ebfe380a.slice/crio-833a3984273bed883e2fba9d8fb89afef3ec7e43eca7d53263b40ab74693c2c0 WatchSource:0}: Error finding container 833a3984273bed883e2fba9d8fb89afef3ec7e43eca7d53263b40ab74693c2c0: Status 404 returned error can't find the container with id 833a3984273bed883e2fba9d8fb89afef3ec7e43eca7d53263b40ab74693c2c0
Oct 31 16:56:29.540192 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:29.540162    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:29.540255 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:29.540224    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:29.540293 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:29.540252    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:30.510964 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:30.510925    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerStarted","Data":"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"}
Oct 31 16:56:30.511462 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:30.510984    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerStarted","Data":"833a3984273bed883e2fba9d8fb89afef3ec7e43eca7d53263b40ab74693c2c0"}
Oct 31 16:56:30.665687 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:30.665634    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podStartSLOduration=2.6656087360000003 podStartE2EDuration="2.665608736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:56:30.663350389 +0000 UTC m=+3235.724925574" watchObservedRunningTime="2026-10-31 16:56:30.665608736 +0000 UTC m=+3235.727183922"
Oct 31 16:56:31.109442 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:31.109410    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:31.114106 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:31.114084    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:31.516355 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:31.516278    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:31.517179 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:31.517161    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:32.943983 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:56:32.943948    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:56:36.640477 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:36.640444    4417 scope.go:122] "RemoveContainer" containerID="dbbc858318a088836b670a4491ea6e553dfd6480cc39e38cafb427283e017fce"
Oct 31 16:56:37.304210 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.304174    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:56:37.309122 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.309101    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.403060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.403025    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-486d5416-dockercfg-jx9dm\""
Oct 31 16:56:37.403060 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.403026    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f191a7\""
Oct 31 16:56:37.406169 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.406113    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:56:37.428053 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.428018    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mwh\" (UniqueName: \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.428264 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.428159    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.470238 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.470198    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:56:37.473708 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.473685    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.529527 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.529499    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fld\" (UniqueName: \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.529684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.529536    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mwh\" (UniqueName: \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.529684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.529561    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.529684 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.529678    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.530285 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.530267    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.530521 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.530503    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:56:37.583061 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.582983    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mwh\" (UniqueName: \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\") pod \"payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.621728 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.621694    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:37.630958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.630938    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fld\" (UniqueName: \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.631040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.630978    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.631603 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.631586    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.688682 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.685632    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fld\" (UniqueName: \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\") pod \"payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.785982 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.785945    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:37.987128 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:37.987098    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:56:37.994181 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:56:37.994153    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96881f7_723a_4ede_bde4_43d158af676f.slice/crio-72b3b8de8eb5e0fc33843f026a2b8c4b9b726cafff9c6db3d3acab6297f1b9c6 WatchSource:0}: Error finding container 72b3b8de8eb5e0fc33843f026a2b8c4b9b726cafff9c6db3d3acab6297f1b9c6: Status 404 returned error can't find the container with id 72b3b8de8eb5e0fc33843f026a2b8c4b9b726cafff9c6db3d3acab6297f1b9c6
Oct 31 16:56:38.152091 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.152007    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:56:38.159584 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:56:38.159556    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7ecf8b_9fdb_4a3b_8b3a_67b453936c7c.slice/crio-70f11ec5da82cfd16ddf6a52b3647b29b51b391bf792ec52493900e7ea0a8e73 WatchSource:0}: Error finding container 70f11ec5da82cfd16ddf6a52b3647b29b51b391bf792ec52493900e7ea0a8e73: Status 404 returned error can't find the container with id 70f11ec5da82cfd16ddf6a52b3647b29b51b391bf792ec52493900e7ea0a8e73
Oct 31 16:56:38.551576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.551541    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" event={"ID":"b96881f7-723a-4ede-bde4-43d158af676f","Type":"ContainerStarted","Data":"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"}
Oct 31 16:56:38.551783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.551734    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:38.551866 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.551779    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" event={"ID":"b96881f7-723a-4ede-bde4-43d158af676f","Type":"ContainerStarted","Data":"72b3b8de8eb5e0fc33843f026a2b8c4b9b726cafff9c6db3d3acab6297f1b9c6"}
Oct 31 16:56:38.553074 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.553048    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" event={"ID":"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c","Type":"ContainerStarted","Data":"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"}
Oct 31 16:56:38.553209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.553085    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" event={"ID":"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c","Type":"ContainerStarted","Data":"70f11ec5da82cfd16ddf6a52b3647b29b51b391bf792ec52493900e7ea0a8e73"}
Oct 31 16:56:38.553209 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.553147    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:38.658282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.658232    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" podStartSLOduration=1.658218489 podStartE2EDuration="1.658218489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:56:38.657320973 +0000 UTC m=+3243.718896159" watchObservedRunningTime="2026-10-31 16:56:38.658218489 +0000 UTC m=+3243.719793699"
Oct 31 16:56:38.761851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:38.761804    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" podStartSLOduration=1.761790211 podStartE2EDuration="1.761790211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:56:38.76106407 +0000 UTC m=+3243.822639255" watchObservedRunningTime="2026-10-31 16:56:38.761790211 +0000 UTC m=+3243.823365397"
Oct 31 16:56:40.512738 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.512698    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:56:40.516422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.516397    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.558818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.558786    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.558968 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.558933    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.559042 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.559026    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qb5\" (UniqueName: \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.559082 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.559056    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.610507 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.610478    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f191a7-serving-cert\""
Oct 31 16:56:40.610675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.610550    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:56:40.610675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.610595    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:56:40.610675 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.610599    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:56:40.610814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.610685    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\""
Oct 31 16:56:40.612987 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.612968    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:56:40.660305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.660277    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qb5\" (UniqueName: \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.660427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.660313    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.660427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.660336    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.660427 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.660417    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.660559 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:56:40.660492    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-serving-cert: secret "maas-api-e2e-worker-single-main-f191a7-serving-cert" not found
Oct 31 16:56:40.660598 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:56:40.660559    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls podName:7902a0a1-c37e-4cfe-b664-3b63d8d2687e nodeName:}" failed. No retries permitted until 2026-10-31 16:56:41.160538667 +0000 UTC m=+3246.222113832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls") pod "maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" (UID: "7902a0a1-c37e-4cfe-b664-3b63d8d2687e") : secret "maas-api-e2e-worker-single-main-f191a7-serving-cert" not found
Oct 31 16:56:40.661105 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.661086    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.662762 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.662746    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:40.713023 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:40.712996    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qb5\" (UniqueName: \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:41.164822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:41.164788    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:41.167254 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:41.167232    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:41.429697 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:41.429596    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:41.795783 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:41.795754    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:56:41.802264 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:56:41.802236    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7902a0a1_c37e_4cfe_b664_3b63d8d2687e.slice/crio-360302d06a11a7f38e17502936656acae184310580397c29546bdf53256f90ac WatchSource:0}: Error finding container 360302d06a11a7f38e17502936656acae184310580397c29546bdf53256f90ac: Status 404 returned error can't find the container with id 360302d06a11a7f38e17502936656acae184310580397c29546bdf53256f90ac
Oct 31 16:56:41.803712 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:41.803690    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:56:42.575261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:42.575219    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" event={"ID":"7902a0a1-c37e-4cfe-b664-3b63d8d2687e","Type":"ContainerStarted","Data":"360302d06a11a7f38e17502936656acae184310580397c29546bdf53256f90ac"}
Oct 31 16:56:43.581822 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:43.581786    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" event={"ID":"7902a0a1-c37e-4cfe-b664-3b63d8d2687e","Type":"ContainerStarted","Data":"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"}
Oct 31 16:56:43.582222 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:43.581871    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:43.689547 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:43.689495    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" podStartSLOduration=2.750368609 podStartE2EDuration="3.689483557s" totalImagesPullingTime="939.114948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:56:43.687480503 +0000 UTC m=+3248.749055691" watchObservedRunningTime="2026-10-31 16:56:43.689483557 +0000 UTC m=+3248.751058742"
Oct 31 16:56:45.049524 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:45.049491    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"]
Oct 31 16:56:45.087126 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:45.087099    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6bhbg"]
Oct 31 16:56:47.048050 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:47.048018    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06831a9-b894-47c2-b592-40a68d8e5aa6" path="/var/lib/kubelet/pods/e06831a9-b894-47c2-b592-40a68d8e5aa6/volumes"
Oct 31 16:56:49.562861 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:49.562825    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:56:49.563421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:49.563334    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:56:49.593376 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:49.593346    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:56:56.044444 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:56.044405    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"]
Oct 31 16:56:56.089183 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:56.089139    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b7msz"]
Oct 31 16:56:57.050405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.050368    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b413afd-cd20-4e60-9f17-288740531e53" path="/var/lib/kubelet/pods/1b413afd-cd20-4e60-9f17-288740531e53/volumes"
Oct 31 16:56:57.668503 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.668479    4417 generic.go:356] "Generic (PLEG): container finished" podID="01757844-146f-4ebe-a426-51f6ebfe380a" containerID="f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d" exitCode=0
Oct 31 16:56:57.668664 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.668552    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerDied","Data":"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"}
Oct 31 16:56:57.668936 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.668924    4417 scope.go:122] "RemoveContainer" containerID="f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"
Oct 31 16:56:57.670062 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.670035    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:57.670137 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.670124    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:57.670176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:57.670159    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 16:56:58.675305 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:58.675272    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerStarted","Data":"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"}
Oct 31 16:56:59.109551 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:59.109513    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:56:59.110604 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:59.110580    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:56:59.110747 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:56:59.110652    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 16:57:00.109131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:00.109097    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:57:00.109648 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:00.109162    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 16:57:01.110085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.110051    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:57:01.110533 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.110111    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 16:57:01.517040 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.516956    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:57:01.787361 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.787276    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:57:01.791058 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.791033    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.886004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.885965    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f191a7\"/\"kube-root-ca.crt\""
Oct 31 16:57:01.886004 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.885978    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f191a7\"/\"default-dockercfg-np6gp\""
Oct 31 16:57:01.886246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.886011    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f191a7\"/\"openshift-service-ca.crt\""
Oct 31 16:57:01.886246 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.886157    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f191a7\"/\"simulator-single-main-f191a7-kserve-self-signed-certs\""
Oct 31 16:57:01.888405 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.888384    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:57:01.889529 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889507    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.889587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889546    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.889587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889573    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.889702 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889688    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.889787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889773    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.889842 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.889801    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbb7\" (UniqueName: \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991028 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.990991    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbb7\" (UniqueName: \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991048    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991096    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991234 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991132    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991294    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991419 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991402    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991490 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991462    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991760    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.991925 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.991771    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.993380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.993357    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:01.993796 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:01.993775    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:02.066525 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.066489    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbb7\" (UniqueName: \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\") pod \"simulator-single-main-f191a7-kserve-864bbb5bd5-6st26\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") " pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:02.102787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.102745    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:02.109173 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.109150    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:57:02.109280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.109200    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 16:57:02.513559 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.513472    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:57:02.519908 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:57:02.519882    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffc86957_3898_4f43_a0f4_a115dea6bf35.slice/crio-436e1a384b174d4b71f7af42ee356afab33fe2ca686d4f88f45f3f57bae896a9 WatchSource:0}: Error finding container 436e1a384b174d4b71f7af42ee356afab33fe2ca686d4f88f45f3f57bae896a9: Status 404 returned error can't find the container with id 436e1a384b174d4b71f7af42ee356afab33fe2ca686d4f88f45f3f57bae896a9
Oct 31 16:57:02.698198 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.698160    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" event={"ID":"ffc86957-3898-4f43-a0f4-a115dea6bf35","Type":"ContainerStarted","Data":"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"}
Oct 31 16:57:02.698417 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.698215    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" event={"ID":"ffc86957-3898-4f43-a0f4-a115dea6bf35","Type":"ContainerStarted","Data":"436e1a384b174d4b71f7af42ee356afab33fe2ca686d4f88f45f3f57bae896a9"}
Oct 31 16:57:02.810415 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:02.810364    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" podStartSLOduration=1.8103523990000001 podStartE2EDuration="1.810352399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:57:02.80937788 +0000 UTC m=+3267.870953066" watchObservedRunningTime="2026-10-31 16:57:02.810352399 +0000 UTC m=+3267.871927625"
Oct 31 16:57:02.973479 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:57:02.973443    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:57:03.103404 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:03.103321    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:03.109723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:03.109701    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:57:03.109792 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:03.109747    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 31 16:57:04.114267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:04.114231    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:57:04.121723 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:04.121696    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:57:12.103402 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:12.103365    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:12.121787 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:12.121758    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:12.765710 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:12.765679    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:57:31.048808 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:31.048777    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"]
Oct 31 16:57:31.088147 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:31.088119    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-zdwch"]
Oct 31 16:57:32.943606 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:57:32.943570    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:57:33.049506 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:33.049475    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b80bf86-c0e9-4b0f-b337-04b65c3b0766" path="/var/lib/kubelet/pods/9b80bf86-c0e9-4b0f-b337-04b65c3b0766/volumes"
Oct 31 16:57:36.662269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:36.662239    4417 scope.go:122] "RemoveContainer" containerID="ec5bcb3d2fd20099247c8d43143c168b2cae1b73364707c4c5fed376de4bc456"
Oct 31 16:57:36.670852 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:36.670828    4417 scope.go:122] "RemoveContainer" containerID="1f40f1e857effff89e603fa1ee74e9cb888f675e7c9f775665046358e05434fd"
Oct 31 16:57:59.049034 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:59.049000    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"]
Oct 31 16:57:59.087704 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:57:59.087667    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-j2rz6"]
Oct 31 16:58:01.049048 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:01.049021    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135825d3-60d4-4515-bfe9-ac2139d441da" path="/var/lib/kubelet/pods/135825d3-60d4-4515-bfe9-ac2139d441da/volumes"
Oct 31 16:58:02.944482 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:02.944445    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:58:03.558435 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:03.558401    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:58:10.591923 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:10.591889    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:58:10.698261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:10.698206    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:58:14.635986 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:14.635956    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:58:14.636398 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:14.636136    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" podUID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" containerName="maas-api" containerID="cri-o://45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d" gracePeriod=30
Oct 31 16:58:15.592897 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:15.592854    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing" containerID="cri-o://c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521" gracePeriod=25
Oct 31 16:58:15.699418 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:15.699368    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing" containerID="cri-o://dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5" gracePeriod=25
Oct 31 16:58:15.984271 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:15.984246    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:58:16.078901 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.078864    4417 generic.go:356] "Generic (PLEG): container finished" podID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" containerID="45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d" exitCode=0
Oct 31 16:58:16.079096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.078931    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"
Oct 31 16:58:16.079096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.078957    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" event={"ID":"7902a0a1-c37e-4cfe-b664-3b63d8d2687e","Type":"ContainerDied","Data":"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"}
Oct 31 16:58:16.079096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.079047    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z" event={"ID":"7902a0a1-c37e-4cfe-b664-3b63d8d2687e","Type":"ContainerDied","Data":"360302d06a11a7f38e17502936656acae184310580397c29546bdf53256f90ac"}
Oct 31 16:58:16.079096 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.079076    4417 scope.go:122] "RemoveContainer" containerID="45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"
Oct 31 16:58:16.087918 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.087893    4417 scope.go:122] "RemoveContainer" containerID="45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"
Oct 31 16:58:16.088180 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:16.088162    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d\": container with ID starting with 45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d not found: ID does not exist" containerID="45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"
Oct 31 16:58:16.088258 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.088188    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d"} err="failed to get container status \"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d\": rpc error: code = NotFound desc = could not find container \"45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d\": container with ID starting with 45d35376c72931acd533609b29f67135c4293bae42afda39ec9e5e38ddbc917d not found: ID does not exist"
Oct 31 16:58:16.110142 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.110081    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\" (UniqueName: \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\") pod \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") "
Oct 31 16:58:16.110274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.110166    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\") pod \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") "
Oct 31 16:58:16.110274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.110215    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\") pod \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") "
Oct 31 16:58:16.110274 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.110243    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") pod \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\" (UID: \"7902a0a1-c37e-4cfe-b664-3b63d8d2687e\") "
Oct 31 16:58:16.110567 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.110533    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca" pod "7902a0a1-c37e-4cfe-b664-3b63d8d2687e" (UID: "7902a0a1-c37e-4cfe-b664-3b63d8d2687e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:16.112257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.112234    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5" pod "7902a0a1-c37e-4cfe-b664-3b63d8d2687e" (UID: "7902a0a1-c37e-4cfe-b664-3b63d8d2687e"). InnerVolumeSpecName "kube-api-access-r2qb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:16.112257 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.112245    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs" pod "7902a0a1-c37e-4cfe-b664-3b63d8d2687e" (UID: "7902a0a1-c37e-4cfe-b664-3b63d8d2687e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:58:16.112389 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.112318    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls" pod "7902a0a1-c37e-4cfe-b664-3b63d8d2687e" (UID: "7902a0a1-c37e-4cfe-b664-3b63d8d2687e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:58:16.211806 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.211774    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:16.212011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.211804    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:16.212011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.211829    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:16.212011 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.211838    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qb5\" (UniqueName: \"kubernetes.io/projected/7902a0a1-c37e-4cfe-b664-3b63d8d2687e-kube-api-access-r2qb5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:16.492070 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.492034    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:58:16.537636 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:16.537595    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f191a7-dcc8bdb9-tc76z"]
Oct 31 16:58:17.048578 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:17.048546    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" path="/var/lib/kubelet/pods/7902a0a1-c37e-4cfe-b664-3b63d8d2687e/volumes"
Oct 31 16:58:18.578380 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.578338    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerName="main" containerID="cri-o://72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174" gracePeriod=45
Oct 31 16:58:18.838146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.838071    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:58:18.939239 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939261    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939319    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939352    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939451 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939403    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\" (UniqueName: \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939463    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\") pod \"ffc86957-3898-4f43-a0f4-a115dea6bf35\" (UID: \"ffc86957-3898-4f43-a0f4-a115dea6bf35\") "
Oct 31 16:58:18.939698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939477    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:18.939698 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939509    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:18.939860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939797    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:18.939860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939814    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:18.939860 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.939833    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:18.942167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.942132    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:18.942281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.942173    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "kube-api-access-dgbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:18.942281 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:18.942211    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs" pod "ffc86957-3898-4f43-a0f4-a115dea6bf35" (UID: "ffc86957-3898-4f43-a0f4-a115dea6bf35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:58:19.040580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.040546    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:19.040580 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.040581    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc86957-3898-4f43-a0f4-a115dea6bf35-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:19.040799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.040596    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgbb7\" (UniqueName: \"kubernetes.io/projected/ffc86957-3898-4f43-a0f4-a115dea6bf35-kube-api-access-dgbb7\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:19.040799 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.040611    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc86957-3898-4f43-a0f4-a115dea6bf35-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:19.096027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.095940    4417 generic.go:356] "Generic (PLEG): container finished" podID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerID="72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174" exitCode=0
Oct 31 16:58:19.096027 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.095992    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"
Oct 31 16:58:19.096249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.095996    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" event={"ID":"ffc86957-3898-4f43-a0f4-a115dea6bf35","Type":"ContainerDied","Data":"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"}
Oct 31 16:58:19.096249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.096101    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" event={"ID":"ffc86957-3898-4f43-a0f4-a115dea6bf35","Type":"ContainerDied","Data":"436e1a384b174d4b71f7af42ee356afab33fe2ca686d4f88f45f3f57bae896a9"}
Oct 31 16:58:19.096249 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.096124    4417 scope.go:122] "RemoveContainer" containerID="72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"
Oct 31 16:58:19.106563 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.106542    4417 scope.go:122] "RemoveContainer" containerID="72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"
Oct 31 16:58:19.107000 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:19.106981    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174\": container with ID starting with 72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174 not found: ID does not exist" containerID="72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"
Oct 31 16:58:19.107057 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.107010    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174"} err="failed to get container status \"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174\": rpc error: code = NotFound desc = could not find container \"72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174\": container with ID starting with 72ae16c3a7b9ec09da994b56e61d69abea48439cabb2552eedb50ece9f2ad174 not found: ID does not exist"
Oct 31 16:58:19.249779 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.249728    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:58:19.256321 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.256284    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26"]
Oct 31 16:58:19.560775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.560739    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:58:19.560775 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.560771    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:58:19.756052 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:19.756004    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-f191a7/simulator-single-main-f191a7-kserve-864bbb5bd5-6st26" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:58:21.049486 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:21.049453    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" path="/var/lib/kubelet/pods/ffc86957-3898-4f43-a0f4-a115dea6bf35/volumes"
Oct 31 16:58:23.494933 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:23.494903    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:58:23.495341 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:23.495105    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy" containerID="cri-o://606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890" gracePeriod=30
Oct 31 16:58:25.073709 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.073677    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"]
Oct 31 16:58:25.119699 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.119663    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zwhfn"]
Oct 31 16:58:25.203863 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.203828    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"]
Oct 31 16:58:25.204280 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204262    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" containerName="maas-api"
Oct 31 16:58:25.204373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204283    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" containerName="maas-api"
Oct 31 16:58:25.204373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204333    4417 cpu_manager.go:414] "containerMap: removing container" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerName="main"
Oct 31 16:58:25.204373 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204342    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerName="main"
Oct 31 16:58:25.204535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204436    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7902a0a1-c37e-4cfe-b664-3b63d8d2687e" containerName="maas-api"
Oct 31 16:58:25.204535 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.204470    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc86957-3898-4f43-a0f4-a115dea6bf35" containerName="main"
Oct 31 16:58:25.208587 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.208567    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.256850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.256798    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:58:25.256850 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.256835    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 16:58:25.257022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.256915    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:58:25.257022 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.256936    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:58:25.264156 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.264132    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"]
Oct 31 16:58:25.299851 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.299826    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.299995 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.299896    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.300093 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.300070    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mbln\" (UniqueName: \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.400832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.400764    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.400832 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.400807    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mbln\" (UniqueName: \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.400989 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.400867    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.401537 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.401516    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.403288 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.403262    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.453112 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.453087    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mbln\" (UniqueName: \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\") pod \"maas-api-revoke-sub-3d4156f1-wcr4b\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.520820 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.520794    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:25.886387 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:25.886352    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"]
Oct 31 16:58:25.893403 ip-10-0-140-84 kubenswrapper[4417]: W1031 16:58:25.893373    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a991b4_913c_4f4e_b14a_69bff5034e5a.slice/crio-e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3 WatchSource:0}: Error finding container e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3: Status 404 returned error can't find the container with id e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3
Oct 31 16:58:26.133025 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:26.132994    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b" event={"ID":"53a991b4-913c-4f4e-b14a-69bff5034e5a","Type":"ContainerStarted","Data":"e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3"}
Oct 31 16:58:27.047422 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:27.047390    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf1c4cb-485d-480f-a86e-bee097feabfb" path="/var/lib/kubelet/pods/5cf1c4cb-485d-480f-a86e-bee097feabfb/volumes"
Oct 31 16:58:27.139973 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:27.139934    4417 generic.go:356] "Generic (PLEG): container finished" podID="53a991b4-913c-4f4e-b14a-69bff5034e5a" containerID="a873b27983507bad436ec1e93eb19e5035a085334ba2f5bbb28aefc4c6145129" exitCode=0
Oct 31 16:58:27.140385 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:27.140054    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b" event={"ID":"53a991b4-913c-4f4e-b14a-69bff5034e5a","Type":"ContainerDied","Data":"a873b27983507bad436ec1e93eb19e5035a085334ba2f5bbb28aefc4c6145129"}
Oct 31 16:58:28.275740 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.275709    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:28.327167 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.327130    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\" (UniqueName: \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\") pod \"53a991b4-913c-4f4e-b14a-69bff5034e5a\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") "
Oct 31 16:58:28.327372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.327177    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\") pod \"53a991b4-913c-4f4e-b14a-69bff5034e5a\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") "
Oct 31 16:58:28.327372 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.327234    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\") pod \"53a991b4-913c-4f4e-b14a-69bff5034e5a\" (UID: \"53a991b4-913c-4f4e-b14a-69bff5034e5a\") "
Oct 31 16:58:28.327594 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.327571    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca" pod "53a991b4-913c-4f4e-b14a-69bff5034e5a" (UID: "53a991b4-913c-4f4e-b14a-69bff5034e5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:28.329432 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.329396    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca" pod "53a991b4-913c-4f4e-b14a-69bff5034e5a" (UID: "53a991b4-913c-4f4e-b14a-69bff5034e5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:58:28.329543 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.329443    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln" pod "53a991b4-913c-4f4e-b14a-69bff5034e5a" (UID: "53a991b4-913c-4f4e-b14a-69bff5034e5a"). InnerVolumeSpecName "kube-api-access-5mbln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:28.428354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.428271    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mbln\" (UniqueName: \"kubernetes.io/projected/53a991b4-913c-4f4e-b14a-69bff5034e5a-kube-api-access-5mbln\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.428354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.428302    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.428354 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.428312    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53a991b4-913c-4f4e-b14a-69bff5034e5a-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.727300 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.727274    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:58:28.832176 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832140    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832195    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832228    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832297    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832371 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832395    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832428    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832445    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:28.832573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832455    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832573 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832529    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\") pod \"01757844-146f-4ebe-a426-51f6ebfe380a\" (UID: \"01757844-146f-4ebe-a426-51f6ebfe380a\") "
Oct 31 16:58:28.832885 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832810    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:28.832949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832919    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:28.832949 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.832933    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:28.833085 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.833059    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.833149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.833097    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.833149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.833113    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.833149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.833126    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.833149 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.833093    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:28.834814 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.834779    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:28.834962 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.834943    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:58:28.835030 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.834968    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:58:28.835030 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.834970    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp" pod "01757844-146f-4ebe-a426-51f6ebfe380a" (UID: "01757844-146f-4ebe-a426-51f6ebfe380a"). InnerVolumeSpecName "kube-api-access-vnbnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:28.934576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.934489    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01757844-146f-4ebe-a426-51f6ebfe380a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.934576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.934518    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01757844-146f-4ebe-a426-51f6ebfe380a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.934576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.934530    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.934576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.934538    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01757844-146f-4ebe-a426-51f6ebfe380a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:28.934576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:28.934549    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbnp\" (UniqueName: \"kubernetes.io/projected/01757844-146f-4ebe-a426-51f6ebfe380a-kube-api-access-vnbnp\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:29.153146 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.153114    4417 generic.go:356] "Generic (PLEG): container finished" podID="01757844-146f-4ebe-a426-51f6ebfe380a" containerID="606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890" exitCode=0
Oct 31 16:58:29.153350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.153162    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"
Oct 31 16:58:29.153350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.153196    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerDied","Data":"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"}
Oct 31 16:58:29.153350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.153246    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt" event={"ID":"01757844-146f-4ebe-a426-51f6ebfe380a","Type":"ContainerDied","Data":"833a3984273bed883e2fba9d8fb89afef3ec7e43eca7d53263b40ab74693c2c0"}
Oct 31 16:58:29.153350 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.153270    4417 scope.go:122] "RemoveContainer" containerID="606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"
Oct 31 16:58:29.154900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.154759    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"
Oct 31 16:58:29.154900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.154783    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b" event={"ID":"53a991b4-913c-4f4e-b14a-69bff5034e5a","Type":"ContainerDied","Data":"e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3"}
Oct 31 16:58:29.154900 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.154833    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88a82c5a6da40f3d05f99ae481cba55727a2d5a8cd8225a09d86e455966fda3"
Oct 31 16:58:29.163261 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.163240    4417 scope.go:122] "RemoveContainer" containerID="f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"
Oct 31 16:58:29.172009 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.171982    4417 scope.go:122] "RemoveContainer" containerID="606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"
Oct 31 16:58:29.172319 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:29.172295    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890\": container with ID starting with 606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890 not found: ID does not exist" containerID="606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"
Oct 31 16:58:29.172452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.172322    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890"} err="failed to get container status \"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890\": rpc error: code = NotFound desc = could not find container \"606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890\": container with ID starting with 606ac1c6644e407a5169071922d94d7f4dbfc8134086f7de81f3e36dccd9c890 not found: ID does not exist"
Oct 31 16:58:29.172452 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.172343    4417 scope.go:122] "RemoveContainer" containerID="f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"
Oct 31 16:58:29.172640 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:29.172599    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d\": container with ID starting with f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d not found: ID does not exist" containerID="f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"
Oct 31 16:58:29.172725 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.172646    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d"} err="failed to get container status \"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d\": rpc error: code = NotFound desc = could not find container \"f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d\": container with ID starting with f01e2d182f1bf848dbe682f3544e6766e24c099e276289a9ad723d8ff7b3e37d not found: ID does not exist"
Oct 31 16:58:29.352320 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.352285    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:58:29.398499 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.398468    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f191a7-openshift-default-58b78bdcdftmvzt"]
Oct 31 16:58:29.560457 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.560420    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:58:29.560674 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:29.560521    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:58:30.041858 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:30.041824    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"]
Oct 31 16:58:30.044745 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:30.044720    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-tbnt6"]
Oct 31 16:58:31.058038 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:31.058004    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" path="/var/lib/kubelet/pods/01757844-146f-4ebe-a426-51f6ebfe380a/volumes"
Oct 31 16:58:31.058599 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:31.058580    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d290c53-c596-43b7-999e-7581f51b073d" path="/var/lib/kubelet/pods/9d290c53-c596-43b7-999e-7581f51b073d/volumes"
Oct 31 16:58:32.943312 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:32.943276    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:58:35.833221 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.833196    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:58:35.900383 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.900355    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\") pod \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") "
Oct 31 16:58:35.900570 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.900475    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\" (UniqueName: \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\") pod \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\" (UID: \"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c\") "
Oct 31 16:58:35.900837 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.900810    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume" pod "1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" (UID: "1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:35.903120 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.903091    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld" pod "1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" (UID: "1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c"). InnerVolumeSpecName "kube-api-access-v2fld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:35.929260 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:35.929237    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:58:36.001375 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.001340    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\" (UniqueName: \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\") pod \"b96881f7-723a-4ede-bde4-43d158af676f\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") "
Oct 31 16:58:36.001517 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.001406    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\") pod \"b96881f7-723a-4ede-bde4-43d158af676f\" (UID: \"b96881f7-723a-4ede-bde4-43d158af676f\") "
Oct 31 16:58:36.001638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.001608    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:36.001638 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.001633    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fld\" (UniqueName: \"kubernetes.io/projected/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c-kube-api-access-v2fld\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:36.001818 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.001796    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume" pod "b96881f7-723a-4ede-bde4-43d158af676f" (UID: "b96881f7-723a-4ede-bde4-43d158af676f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:58:36.003440 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.003424    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh" pod "b96881f7-723a-4ede-bde4-43d158af676f" (UID: "b96881f7-723a-4ede-bde4-43d158af676f"). InnerVolumeSpecName "kube-api-access-54mwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:58:36.102232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.102198    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mwh\" (UniqueName: \"kubernetes.io/projected/b96881f7-723a-4ede-bde4-43d158af676f-kube-api-access-54mwh\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:36.102232 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.102230    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96881f7-723a-4ede-bde4-43d158af676f-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:58:36.190080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.189989    4417 generic.go:356] "Generic (PLEG): container finished" podID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerID="c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521" exitCode=0
Oct 31 16:58:36.190080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.190035    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"
Oct 31 16:58:36.190080 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.190071    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" event={"ID":"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c","Type":"ContainerDied","Data":"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"}
Oct 31 16:58:36.190351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.190122    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch" event={"ID":"1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c","Type":"ContainerDied","Data":"70f11ec5da82cfd16ddf6a52b3647b29b51b391bf792ec52493900e7ea0a8e73"}
Oct 31 16:58:36.190351 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.190147    4417 scope.go:122] "RemoveContainer" containerID="c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"
Oct 31 16:58:36.191518 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.191496    4417 generic.go:356] "Generic (PLEG): container finished" podID="b96881f7-723a-4ede-bde4-43d158af676f" containerID="dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5" exitCode=0
Oct 31 16:58:36.191671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.191543    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"
Oct 31 16:58:36.191671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.191570    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" event={"ID":"b96881f7-723a-4ede-bde4-43d158af676f","Type":"ContainerDied","Data":"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"}
Oct 31 16:58:36.191671 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.191635    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb" event={"ID":"b96881f7-723a-4ede-bde4-43d158af676f","Type":"ContainerDied","Data":"72b3b8de8eb5e0fc33843f026a2b8c4b9b726cafff9c6db3d3acab6297f1b9c6"}
Oct 31 16:58:36.199420 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.199294    4417 scope.go:122] "RemoveContainer" containerID="c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"
Oct 31 16:58:36.199661 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:36.199628    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521\": container with ID starting with c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521 not found: ID does not exist" containerID="c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"
Oct 31 16:58:36.199786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.199672    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521"} err="failed to get container status \"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521\": rpc error: code = NotFound desc = could not find container \"c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521\": container with ID starting with c98c59431d63bca15c12644ceb31949ee033e7ba319367528eaf696087297521 not found: ID does not exist"
Oct 31 16:58:36.199786 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.199696    4417 scope.go:122] "RemoveContainer" containerID="dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"
Oct 31 16:58:36.207646 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.207627    4417 scope.go:122] "RemoveContainer" containerID="dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"
Oct 31 16:58:36.207883 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:58:36.207868    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5\": container with ID starting with dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5 not found: ID does not exist" containerID="dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"
Oct 31 16:58:36.207937 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.207888    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5"} err="failed to get container status \"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5\": rpc error: code = NotFound desc = could not find container \"dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5\": container with ID starting with dee8629346053c51066150998060ca999818caac16df5475b7a1b56da10465c5 not found: ID does not exist"
Oct 31 16:58:36.340078 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.340047    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:58:36.342940 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.342917    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f191a7-55d4d6pcxb"]
Oct 31 16:58:36.441676 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.441591    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:58:36.444116 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.444093    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f191a7-66c4865c8zsdch"]
Oct 31 16:58:36.695134 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.695042    4417 scope.go:122] "RemoveContainer" containerID="68183aa18d25de71b55d6f48d2d526ebe178a64748dd6f7cb59ff038a64ac268"
Oct 31 16:58:36.703421 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.703402    4417 scope.go:122] "RemoveContainer" containerID="b527c9cbe5518a2ac08c023ffae6c7d675ecd32c2b32e8f21cc782dcf42dd096"
Oct 31 16:58:36.711267 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:36.711098    4417 scope.go:122] "RemoveContainer" containerID="0e328d0f55826cec0a0e4c2b660e789b34d17c7167b43df09a1e8023bf0a910f"
Oct 31 16:58:37.048193 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:37.048160    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" path="/var/lib/kubelet/pods/1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c/volumes"
Oct 31 16:58:37.048877 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:37.048746    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96881f7-723a-4ede-bde4-43d158af676f" path="/var/lib/kubelet/pods/b96881f7-723a-4ede-bde4-43d158af676f/volumes"
Oct 31 16:58:54.042102 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:54.042019    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"]
Oct 31 16:58:54.087958 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:54.087927    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbkc4"]
Oct 31 16:58:55.048211 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:55.048177    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90bc8eb-7ee2-4711-804a-b3a75a797273" path="/var/lib/kubelet/pods/c90bc8eb-7ee2-4711-804a-b3a75a797273/volumes"
Oct 31 16:58:59.813576 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:59.813539    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:58:59.814454 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:58:59.813790    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" podUID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" containerName="manager" containerID="cri-o://1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7" gracePeriod=10
Oct 31 16:59:00.066532 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.066473    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:59:00.172132 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.172104    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\") pod \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") "
Oct 31 16:59:00.172308 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.172244    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\" (UniqueName: \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\") pod \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\" (UID: \"e10d41d0-e0d5-44ee-8c48-61aae8a54225\") "
Oct 31 16:59:00.172519 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.172492    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume" pod "e10d41d0-e0d5-44ee-8c48-61aae8a54225" (UID: "e10d41d0-e0d5-44ee-8c48-61aae8a54225"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:59:00.174596 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.174571    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb" pod "e10d41d0-e0d5-44ee-8c48-61aae8a54225" (UID: "e10d41d0-e0d5-44ee-8c48-61aae8a54225"). InnerVolumeSpecName "kube-api-access-lc4gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:59:00.273269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.273232    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4gb\" (UniqueName: \"kubernetes.io/projected/e10d41d0-e0d5-44ee-8c48-61aae8a54225-kube-api-access-lc4gb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:59:00.273269 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.273262    4417 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e10d41d0-e0d5-44ee-8c48-61aae8a54225-extensions-socket-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 16:59:00.331585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.331464    4417 generic.go:356] "Generic (PLEG): container finished" podID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" containerID="1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7" exitCode=0
Oct 31 16:59:00.331585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.331513    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"
Oct 31 16:59:00.331585 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.331545    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" event={"ID":"e10d41d0-e0d5-44ee-8c48-61aae8a54225","Type":"ContainerDied","Data":"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"}
Oct 31 16:59:00.331895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.331604    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6" event={"ID":"e10d41d0-e0d5-44ee-8c48-61aae8a54225","Type":"ContainerDied","Data":"84fb545db50a2d6885a5751d285db2bf2f9deb7272bf949264ac88cc4d73611e"}
Oct 31 16:59:00.331895 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.331653    4417 scope.go:122] "RemoveContainer" containerID="1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"
Oct 31 16:59:00.342104 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.342085    4417 scope.go:122] "RemoveContainer" containerID="1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"
Oct 31 16:59:00.342358 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:59:00.342341    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7\": container with ID starting with 1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7 not found: ID does not exist" containerID="1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"
Oct 31 16:59:00.342413 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.342366    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7"} err="failed to get container status \"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7\": rpc error: code = NotFound desc = could not find container \"1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7\": container with ID starting with 1ad84a0be69593c38a23bd420f9236c0afa6a248a898c9060d6c687ec90914b7 not found: ID does not exist"
Oct 31 16:59:00.482515 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.482484    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:59:00.486340 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:00.486312    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qzfn6"]
Oct 31 16:59:01.047131 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:01.047089    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" path="/var/lib/kubelet/pods/e10d41d0-e0d5-44ee-8c48-61aae8a54225/volumes"
Oct 31 16:59:02.943489 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:59:02.943455    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:59:32.943149 ip-10-0-140-84 kubenswrapper[4417]: E1031 16:59:32.943113    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 16:59:36.753282 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:36.753247    4417 scope.go:122] "RemoveContainer" containerID="45e9fb0d0b245aa1e7ee773c61580030e50d492ecdff0d4c4c02ae5e7be1e074"
Oct 31 16:59:36.761315 ip-10-0-140-84 kubenswrapper[4417]: I1031 16:59:36.761297    4417 scope.go:122] "RemoveContainer" containerID="a536e35604714b032bf9a6e04a8c6cd67a823a3070cd6036f7c9083b06e2e6d9"
Oct 31 17:00:00.156651 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.156600    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:00.157159 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157139    4417 cpu_manager.go:414] "containerMap: removing container" podUID="53a991b4-913c-4f4e-b14a-69bff5034e5a" containerName="revoke-keys"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157164    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a991b4-913c-4f4e-b14a-69bff5034e5a" containerName="revoke-keys"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157185    4417 cpu_manager.go:414] "containerMap: removing container" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157194    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157208    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157216    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157235    4417 cpu_manager.go:414] "containerMap: removing container" podUID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" containerName="manager"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157243    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" containerName="manager"
Oct 31 17:00:00.157259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157260    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157269    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157367    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a991b4-913c-4f4e-b14a-69bff5034e5a" containerName="revoke-keys"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157380    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157391    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96881f7-723a-4ede-bde4-43d158af676f" containerName="payload-pre-processing"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157401    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10d41d0-e0d5-44ee-8c48-61aae8a54225" containerName="manager"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157412    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:00.157718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.157422    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7ecf8b-9fdb-4a3b-8b3a-67b453936c7c" containerName="payload-processing"
Oct 31 17:00:00.160751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.160728    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.201936 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.201909    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.202068 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.201978    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctbf\" (UniqueName: \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.209057 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.209030    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:00:00.254890 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.254857    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:00:00.255032 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.254914    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:00:00.256788 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.256769    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:00.302843 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.302810    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctbf\" (UniqueName: \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.303040 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.302875    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.303484 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.303463    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.355040 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.355013    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctbf\" (UniqueName: \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\") pod \"maas-api-key-cleanup-29891100-xx59m\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.473586 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.473520    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:00.837975 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:00.837942    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:00.845072 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:00:00.845045    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411ef5c3_19da_4ab2_868d_a361892db1f6.slice/crio-adfc41d50cc74f3d660fac8253524a4c497d20d649dff8cc960d1f23a5183fdd WatchSource:0}: Error finding container adfc41d50cc74f3d660fac8253524a4c497d20d649dff8cc960d1f23a5183fdd: Status 404 returned error can't find the container with id adfc41d50cc74f3d660fac8253524a4c497d20d649dff8cc960d1f23a5183fdd
Oct 31 17:00:01.602551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:01.602514    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerStarted","Data":"8a4a26da553f35f56bce40cc1d682007c939be8ed764aa09415fa476dadb80a2"}
Oct 31 17:00:01.602986 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:01.602568    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerStarted","Data":"adfc41d50cc74f3d660fac8253524a4c497d20d649dff8cc960d1f23a5183fdd"}
Oct 31 17:00:01.707568 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:01.707510    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" podStartSLOduration=1.707496269 podStartE2EDuration="1.707496269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:00:01.706548938 +0000 UTC m=+3446.768124124" watchObservedRunningTime="2026-10-31 17:00:01.707496269 +0000 UTC m=+3446.769071456"
Oct 31 17:00:02.609110 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:02.609077    4417 generic.go:356] "Generic (PLEG): container finished" podID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerID="8a4a26da553f35f56bce40cc1d682007c939be8ed764aa09415fa476dadb80a2" exitCode=60
Oct 31 17:00:02.609522 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:02.609167    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerDied","Data":"8a4a26da553f35f56bce40cc1d682007c939be8ed764aa09415fa476dadb80a2"}
Oct 31 17:00:02.609522 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:02.609355    4417 scope.go:122] "RemoveContainer" containerID="8a4a26da553f35f56bce40cc1d682007c939be8ed764aa09415fa476dadb80a2"
Oct 31 17:00:02.943309 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:00:02.943276    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:00:03.450988 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.450957    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"]
Oct 31 17:00:03.451334 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.451322    4417 cpu_manager.go:414] "containerMap: removing container" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:03.451383 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.451336    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01757844-146f-4ebe-a426-51f6ebfe380a" containerName="istio-proxy"
Oct 31 17:00:03.454435 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.454418    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.506955 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.506924    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qkljv\""
Oct 31 17:00:03.534551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.534518    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace48c49-3306-4f3d-a16a-4f90922eea83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.534791 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.534597    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbjq\" (UniqueName: \"kubernetes.io/projected/ace48c49-3306-4f3d-a16a-4f90922eea83-kube-api-access-wjbjq\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.564429 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.564387    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"]
Oct 31 17:00:03.615281 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.615192    4417 generic.go:356] "Generic (PLEG): container finished" podID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerID="a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c" exitCode=60
Oct 31 17:00:03.615772 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.615277    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerDied","Data":"a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c"}
Oct 31 17:00:03.615772 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.615340    4417 scope.go:122] "RemoveContainer" containerID="8a4a26da553f35f56bce40cc1d682007c939be8ed764aa09415fa476dadb80a2"
Oct 31 17:00:03.615772 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.615558    4417 scope.go:122] "RemoveContainer" containerID="a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c"
Oct 31 17:00:03.615946 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:00:03.615852    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891100-xx59m_odh-ai-gateway-infra(411ef5c3-19da-4ab2-868d-a361892db1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6"
Oct 31 17:00:03.635552 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.635526    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace48c49-3306-4f3d-a16a-4f90922eea83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.635714 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.635594    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbjq\" (UniqueName: \"kubernetes.io/projected/ace48c49-3306-4f3d-a16a-4f90922eea83-kube-api-access-wjbjq\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.635934 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.635911    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ace48c49-3306-4f3d-a16a-4f90922eea83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.688831 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.688803    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbjq\" (UniqueName: \"kubernetes.io/projected/ace48c49-3306-4f3d-a16a-4f90922eea83-kube-api-access-wjbjq\") pod \"kuadrant-operator-controller-manager-678d87766f-m5whb\" (UID: \"ace48c49-3306-4f3d-a16a-4f90922eea83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:03.795857 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:03.795824    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:04.216208 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.216176    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"]
Oct 31 17:00:04.223524 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:00:04.223495    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace48c49_3306_4f3d_a16a_4f90922eea83.slice/crio-97ad451e203886db7691311c2321d9b22eee767fc824d76e3e9c01e33efb5776 WatchSource:0}: Error finding container 97ad451e203886db7691311c2321d9b22eee767fc824d76e3e9c01e33efb5776: Status 404 returned error can't find the container with id 97ad451e203886db7691311c2321d9b22eee767fc824d76e3e9c01e33efb5776
Oct 31 17:00:04.621814 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.621774    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb" event={"ID":"ace48c49-3306-4f3d-a16a-4f90922eea83","Type":"ContainerStarted","Data":"5b6cc77a7c093b253da1ea44e1f2894e2fae0232ea612c975eb28cf23154ecbd"}
Oct 31 17:00:04.622290 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.621840    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb" event={"ID":"ace48c49-3306-4f3d-a16a-4f90922eea83","Type":"ContainerStarted","Data":"97ad451e203886db7691311c2321d9b22eee767fc824d76e3e9c01e33efb5776"}
Oct 31 17:00:04.622290 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.621959    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:04.623609 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.623592    4417 scope.go:122] "RemoveContainer" containerID="a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c"
Oct 31 17:00:04.623834 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:00:04.623815    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891100-xx59m_odh-ai-gateway-infra(411ef5c3-19da-4ab2-868d-a361892db1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6"
Oct 31 17:00:04.730259 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:04.730210    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb" podStartSLOduration=1.730196874 podStartE2EDuration="1.730196874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:00:04.729156217 +0000 UTC m=+3449.790731403" watchObservedRunningTime="2026-10-31 17:00:04.730196874 +0000 UTC m=+3449.791772059"
Oct 31 17:00:15.630701 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:15.630674    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m5whb"
Oct 31 17:00:16.042033 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:16.042008    4417 scope.go:122] "RemoveContainer" containerID="a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c"
Oct 31 17:00:16.679346 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:16.679310    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerStarted","Data":"8d4bd5eb28fc6e334fff68e00679881ff6e623644f43410c47e35708d50b16f4"}
Oct 31 17:00:17.685231 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:17.685198    4417 generic.go:356] "Generic (PLEG): container finished" podID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerID="8d4bd5eb28fc6e334fff68e00679881ff6e623644f43410c47e35708d50b16f4" exitCode=60
Oct 31 17:00:17.685735 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:17.685268    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerDied","Data":"8d4bd5eb28fc6e334fff68e00679881ff6e623644f43410c47e35708d50b16f4"}
Oct 31 17:00:17.685735 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:17.685351    4417 scope.go:122] "RemoveContainer" containerID="a219b8c696610bfa90d73331a2762f3301de81266321110c8eef98206e89505c"
Oct 31 17:00:17.685735 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:17.685636    4417 scope.go:122] "RemoveContainer" containerID="8d4bd5eb28fc6e334fff68e00679881ff6e623644f43410c47e35708d50b16f4"
Oct 31 17:00:17.685879 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:00:17.685853    4417 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891100-xx59m_odh-ai-gateway-infra(411ef5c3-19da-4ab2-868d-a361892db1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6"
Oct 31 17:00:17.800432 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:17.800401    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:18.819454 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:18.819434    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:18.979286 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:18.979210    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\") pod \"411ef5c3-19da-4ab2-868d-a361892db1f6\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") "
Oct 31 17:00:18.979286 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:18.979275    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\" (UniqueName: \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\") pod \"411ef5c3-19da-4ab2-868d-a361892db1f6\" (UID: \"411ef5c3-19da-4ab2-868d-a361892db1f6\") "
Oct 31 17:00:18.979569 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:18.979546    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca" pod "411ef5c3-19da-4ab2-868d-a361892db1f6" (UID: "411ef5c3-19da-4ab2-868d-a361892db1f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:00:18.981353 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:18.981332    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf" pod "411ef5c3-19da-4ab2-868d-a361892db1f6" (UID: "411ef5c3-19da-4ab2-868d-a361892db1f6"). InnerVolumeSpecName "kube-api-access-fctbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:00:19.080855 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.080816    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/411ef5c3-19da-4ab2-868d-a361892db1f6-maas-api-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:00:19.080855 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.080852    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctbf\" (UniqueName: \"kubernetes.io/projected/411ef5c3-19da-4ab2-868d-a361892db1f6-kube-api-access-fctbf\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:00:19.696499 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.696469    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"
Oct 31 17:00:19.696702 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.696473    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m" event={"ID":"411ef5c3-19da-4ab2-868d-a361892db1f6","Type":"ContainerDied","Data":"adfc41d50cc74f3d660fac8253524a4c497d20d649dff8cc960d1f23a5183fdd"}
Oct 31 17:00:19.696702 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.696601    4417 scope.go:122] "RemoveContainer" containerID="8d4bd5eb28fc6e334fff68e00679881ff6e623644f43410c47e35708d50b16f4"
Oct 31 17:00:19.845761 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.845731    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:19.891597 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:19.891571    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891100-xx59m"]
Oct 31 17:00:21.047763 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:21.047726    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" path="/var/lib/kubelet/pods/411ef5c3-19da-4ab2-868d-a361892db1f6/volumes"
Oct 31 17:00:27.042812 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:27.042776    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"]
Oct 31 17:00:27.118530 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:27.118497    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-8bt7t"]
Oct 31 17:00:27.998962 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:27.998913    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-dx8zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:00:27.999263 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:27.999237    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-dx8zr" podUID="ec81237f-b4d2-4591-8e0d-43c7a20df439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:00:29.047913 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.047883    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c" path="/var/lib/kubelet/pods/7d1e9742-9ebd-4dc5-9b9b-0f1e1d58db0c/volumes"
Oct 31 17:00:29.307470 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.307378    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"]
Oct 31 17:00:29.307860 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.307842    4417 cpu_manager.go:414] "containerMap: removing container" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.307963 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.307862    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.307963 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.307924    4417 cpu_manager.go:414] "containerMap: removing container" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.307963 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.307933    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.308119 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.308015    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.308119 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.308031    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.308119 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.308041    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:29.312673 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.312650    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.360881 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.360860    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:00:29.406396 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.406366    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:00:29.406573 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.406398    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:00:29.406573 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.406485    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:00:29.408258 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.408231    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"]
Oct 31 17:00:29.410542 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.410515    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.410660 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.410571    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crb5\" (UniqueName: \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.410814 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.410792    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.511606 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.511572    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.511787 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.511670    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.511787 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.511724    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crb5\" (UniqueName: \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.512430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.512400    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.514087 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.514062    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.564055 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.563998    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crb5\" (UniqueName: \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\") pod \"maas-api-revoke-sub-d576e68b-gjn2v\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.625898 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.625874    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:29.991828 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:29.991745    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"]
Oct 31 17:00:29.999140 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:00:29.999102    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7c8b83_c7ea_4be0_bab7_bd7fce22c8df.slice/crio-bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8 WatchSource:0}: Error finding container bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8: Status 404 returned error can't find the container with id bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8
Oct 31 17:00:30.777196 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:30.777168    4417 generic.go:356] "Generic (PLEG): container finished" podID="7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" containerID="885b23aaf733e51bac6b09f2c278ed77247d5f7c042761b066014e4b057e700b" exitCode=0
Oct 31 17:00:30.777545 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:30.777262    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v" event={"ID":"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df","Type":"ContainerDied","Data":"885b23aaf733e51bac6b09f2c278ed77247d5f7c042761b066014e4b057e700b"}
Oct 31 17:00:30.777545 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:30.777328    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v" event={"ID":"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df","Type":"ContainerStarted","Data":"bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8"}
Oct 31 17:00:31.917274 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:31.917251    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:32.034814 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.034728    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\" (UniqueName: \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\") pod \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") "
Oct 31 17:00:32.034967 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.034858    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\") pod \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") "
Oct 31 17:00:32.034967 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.034896    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\") pod \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\" (UID: \"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df\") "
Oct 31 17:00:32.035545 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.035517    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca" pod "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" (UID: "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:00:32.036988 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.036961    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5" pod "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" (UID: "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df"). InnerVolumeSpecName "kube-api-access-6crb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:00:32.037175 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.037157    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca" pod "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" (UID: "7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:00:32.135988 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.135967    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6crb5\" (UniqueName: \"kubernetes.io/projected/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-kube-api-access-6crb5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:00:32.135988 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.135988    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:00:32.136138 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.135999    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:00:32.787860 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.787832    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"
Oct 31 17:00:32.787860 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.787852    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v" event={"ID":"7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df","Type":"ContainerDied","Data":"bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8"}
Oct 31 17:00:32.788073 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:32.787896    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb02002f32ffd33056c0608dbfd0925e67d3b6fbaa277f6ce480abadb15e14d8"
Oct 31 17:00:32.943686 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:00:32.943653    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:00:36.786026 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:36.785981    4417 scope.go:122] "RemoveContainer" containerID="2f63c40099e9e828c756825299764dfafbf6648dd2ab077090e34c0831c18393"
Oct 31 17:00:36.796279 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:36.796232    4417 scope.go:122] "RemoveContainer" containerID="db3774a55b4612f8f73a65241ca7f2e4e7479ba237ea66c96785d505fff85d4b"
Oct 31 17:00:36.804862 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:36.804846    4417 scope.go:122] "RemoveContainer" containerID="1fa21593e45d906e91aa9ba291c804cde81321aee5fec6d20983b2fee6794691"
Oct 31 17:00:38.041967 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:38.041928    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"]
Oct 31 17:00:38.087808 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:38.087776    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-m42g8"]
Oct 31 17:00:39.047248 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:39.047217    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35523db-8577-43af-8c8b-975b929f76d9" path="/var/lib/kubelet/pods/d35523db-8577-43af-8c8b-975b929f76d9/volumes"
Oct 31 17:00:58.338184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338144    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"]
Oct 31 17:00:58.338713 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338557    4417 cpu_manager.go:414] "containerMap: removing container" podUID="7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" containerName="revoke-keys"
Oct 31 17:00:58.338713 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338568    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" containerName="revoke-keys"
Oct 31 17:00:58.338713 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338582    4417 cpu_manager.go:414] "containerMap: removing container" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:58.338713 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338587    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411ef5c3-19da-4ab2-868d-a361892db1f6" containerName="cleanup"
Oct 31 17:00:58.338713 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.338655    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" containerName="revoke-keys"
Oct 31 17:00:58.342191 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.342162    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.421816 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.421781    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhjb\" (UniqueName: \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.421962 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.421909    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.422016 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.421996    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.436544 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.436519    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:00:58.436711 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.436562    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:00:58.436860 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.436844    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:00:58.436988 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.436972    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:00:58.438836 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.438816    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"]
Oct 31 17:00:58.522974 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.522943    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.522974 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.522976    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhjb\" (UniqueName: \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.523167 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.523033    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.523701 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.523680    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.525282 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.525264    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.576117 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.576082    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhjb\" (UniqueName: \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\") pod \"maas-api-revoke-sub-d2bc93bc-6wb2c\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:58.655204 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:58.655132    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:00:59.023140 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:59.023060    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"]
Oct 31 17:00:59.030891 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:00:59.030858    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac8910f_ebcf_4680_8164_db66ff974b13.slice/crio-42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36 WatchSource:0}: Error finding container 42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36: Status 404 returned error can't find the container with id 42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36
Oct 31 17:00:59.948298 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:59.948268    4417 generic.go:356] "Generic (PLEG): container finished" podID="3ac8910f-ebcf-4680-8164-db66ff974b13" containerID="be0eda95ed89198fdb84e4f95fc34648678f8106f77ce08735c57bcd0fc1bfd2" exitCode=0
Oct 31 17:00:59.948676 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:59.948316    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c" event={"ID":"3ac8910f-ebcf-4680-8164-db66ff974b13","Type":"ContainerDied","Data":"be0eda95ed89198fdb84e4f95fc34648678f8106f77ce08735c57bcd0fc1bfd2"}
Oct 31 17:00:59.948676 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:00:59.948375    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c" event={"ID":"3ac8910f-ebcf-4680-8164-db66ff974b13","Type":"ContainerStarted","Data":"42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36"}
Oct 31 17:01:01.095919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.095892    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:01:01.250786 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.250688    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\" (UniqueName: \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\") pod \"3ac8910f-ebcf-4680-8164-db66ff974b13\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") "
Oct 31 17:01:01.250786 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.250771    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\") pod \"3ac8910f-ebcf-4680-8164-db66ff974b13\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") "
Oct 31 17:01:01.251044 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.250903    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\") pod \"3ac8910f-ebcf-4680-8164-db66ff974b13\" (UID: \"3ac8910f-ebcf-4680-8164-db66ff974b13\") "
Oct 31 17:01:01.251173 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.251146    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca" pod "3ac8910f-ebcf-4680-8164-db66ff974b13" (UID: "3ac8910f-ebcf-4680-8164-db66ff974b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:01:01.251246 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.251231    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:01:01.253089 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.253054    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca" pod "3ac8910f-ebcf-4680-8164-db66ff974b13" (UID: "3ac8910f-ebcf-4680-8164-db66ff974b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:01:01.253209 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.253076    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb" pod "3ac8910f-ebcf-4680-8164-db66ff974b13" (UID: "3ac8910f-ebcf-4680-8164-db66ff974b13"). InnerVolumeSpecName "kube-api-access-mmhjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:01:01.352128 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.352092    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmhjb\" (UniqueName: \"kubernetes.io/projected/3ac8910f-ebcf-4680-8164-db66ff974b13-kube-api-access-mmhjb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:01:01.352128 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.352124    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac8910f-ebcf-4680-8164-db66ff974b13-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:01:01.958636 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.958594    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"
Oct 31 17:01:01.958636 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.958604    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c" event={"ID":"3ac8910f-ebcf-4680-8164-db66ff974b13","Type":"ContainerDied","Data":"42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36"}
Oct 31 17:01:01.958867 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:01.958683    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b9aa9e78ce24b7928746b6d447b32c065cfe67b313f073094f5c7422bbca36"
Oct 31 17:01:02.943317 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:01:02.943282    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:01:11.376011 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.375974    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:01:11.376561 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.376543    4417 cpu_manager.go:414] "containerMap: removing container" podUID="3ac8910f-ebcf-4680-8164-db66ff974b13" containerName="revoke-keys"
Oct 31 17:01:11.376671 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.376562    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac8910f-ebcf-4680-8164-db66ff974b13" containerName="revoke-keys"
Oct 31 17:01:11.376747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.376676    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac8910f-ebcf-4680-8164-db66ff974b13" containerName="revoke-keys"
Oct 31 17:01:11.379723 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.379702    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.436913 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.436877    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437108 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.436927    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437108 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.436963    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437108 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437047    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437108 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437070    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437108 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437101    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437392 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437149    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437392 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437200    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.437392 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.437231    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.477903 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.477875    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a7bea2-openshift-default-dockercfg-qk8mj\""
Oct 31 17:01:11.484133 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.484108    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:01:11.538805 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538759    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539013 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538812    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539013 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538848    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539013 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538901    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539013 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538951    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539013 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.538980    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539337 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.539020    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539337 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.539053    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539337 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.539093    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.539949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.539925    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.540229 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.540211    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.540844 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.540798    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.541249 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.541225    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.541451 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.541432    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.546124 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.546098    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.549233 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.547096    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.593110 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.593074    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.593275 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.593134    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\") pod \"e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") " pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:11.692491 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:11.692396    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:12.113059 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:12.113025    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:01:12.120085 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:01:12.120057    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc02ffcd_8ddd_4093_8d0e_779161a7c787.slice/crio-0351a7dfe37eabae343fffe50913a6d45f24c38b3cbfacbd1abe4abe2247df6f WatchSource:0}: Error finding container 0351a7dfe37eabae343fffe50913a6d45f24c38b3cbfacbd1abe4abe2247df6f: Status 404 returned error can't find the container with id 0351a7dfe37eabae343fffe50913a6d45f24c38b3cbfacbd1abe4abe2247df6f
Oct 31 17:01:12.122327 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:12.122296    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:12.122407 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:12.122362    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:12.122407 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:12.122390    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:13.022260 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:13.022223    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerStarted","Data":"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"}
Oct 31 17:01:13.022655 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:13.022281    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerStarted","Data":"0351a7dfe37eabae343fffe50913a6d45f24c38b3cbfacbd1abe4abe2247df6f"}
Oct 31 17:01:13.177289 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:13.177225    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podStartSLOduration=2.177211572 podStartE2EDuration="2.177211572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:01:13.175254423 +0000 UTC m=+3518.236829610" watchObservedRunningTime="2026-10-31 17:01:13.177211572 +0000 UTC m=+3518.238786752"
Oct 31 17:01:13.693631 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:13.693583    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:13.698222 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:13.698199    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:14.027162 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:14.027072    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:14.027974 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:14.027956    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:19.754338 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.754304    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:01:19.757895 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.757870    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.805822 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.805798    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a7bea2\""
Oct 31 17:01:19.818463 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.818435    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.818612 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.818517    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5hv\" (UniqueName: \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.851943 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.851912    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4b070007-dockercfg-nchdx\""
Oct 31 17:01:19.855242 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.855213    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:01:19.914938 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.914907    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:01:19.918383 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.918361    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:19.919230 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.919204    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.919341 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.919299    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5hv\" (UniqueName: \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.920008 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.919988    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.971340 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.971307    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5hv\" (UniqueName: \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\") pod \"payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:19.974883 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:19.974856    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:01:20.021022 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.020941    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhjm\" (UniqueName: \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.021022 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.020997    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.070843 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.070817    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:20.122283 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.122255    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhjm\" (UniqueName: \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.122441 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.122310    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.127753 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.123351    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.175262 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.175216    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhjm\" (UniqueName: \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\") pod \"payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.230724 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.230695    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:20.437181 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.437151    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:01:20.444170 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:01:20.444145    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4271107_57c8_4b2a_879a_aa282b84b492.slice/crio-44c05bafa40764acfc8075717a34ca8e641c026cad6f8f63b23e551876928e4c WatchSource:0}: Error finding container 44c05bafa40764acfc8075717a34ca8e641c026cad6f8f63b23e551876928e4c: Status 404 returned error can't find the container with id 44c05bafa40764acfc8075717a34ca8e641c026cad6f8f63b23e551876928e4c
Oct 31 17:01:20.597652 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:20.597612    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:01:20.605738 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:01:20.605703    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145ca58b_ed57_49d2_ab0b_760e4a7cc28e.slice/crio-b288be9672332e4bd1093c058cb77373fa748eef23dd5f9464b8a9fb9f150e70 WatchSource:0}: Error finding container b288be9672332e4bd1093c058cb77373fa748eef23dd5f9464b8a9fb9f150e70: Status 404 returned error can't find the container with id b288be9672332e4bd1093c058cb77373fa748eef23dd5f9464b8a9fb9f150e70
Oct 31 17:01:21.060745 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.060711    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" event={"ID":"145ca58b-ed57-49d2-ab0b-760e4a7cc28e","Type":"ContainerStarted","Data":"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"}
Oct 31 17:01:21.061171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.060774    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" event={"ID":"145ca58b-ed57-49d2-ab0b-760e4a7cc28e","Type":"ContainerStarted","Data":"b288be9672332e4bd1093c058cb77373fa748eef23dd5f9464b8a9fb9f150e70"}
Oct 31 17:01:21.061171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.060805    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:21.062285 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.062257    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" event={"ID":"d4271107-57c8-4b2a-879a-aa282b84b492","Type":"ContainerStarted","Data":"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"}
Oct 31 17:01:21.062387 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.062308    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:21.062387 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.062324    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" event={"ID":"d4271107-57c8-4b2a-879a-aa282b84b492","Type":"ContainerStarted","Data":"44c05bafa40764acfc8075717a34ca8e641c026cad6f8f63b23e551876928e4c"}
Oct 31 17:01:21.165857 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.165817    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" podStartSLOduration=2.165805081 podStartE2EDuration="2.165805081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:01:21.165209143 +0000 UTC m=+3526.226784329" watchObservedRunningTime="2026-10-31 17:01:21.165805081 +0000 UTC m=+3526.227380271"
Oct 31 17:01:21.270509 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:21.270464    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" podStartSLOduration=2.270451426 podStartE2EDuration="2.270451426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:01:21.26983159 +0000 UTC m=+3526.331406777" watchObservedRunningTime="2026-10-31 17:01:21.270451426 +0000 UTC m=+3526.332026613"
Oct 31 17:01:23.014901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.014869    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:01:23.018737 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.018711    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.050040 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.050008    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7m5\" (UniqueName: \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.050235 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.050079    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.050235 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.050171    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.050326 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.050295    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.067131 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.067087    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a7bea2-serving-cert\""
Oct 31 17:01:23.112887 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.112860    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\""
Oct 31 17:01:23.113086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.112890    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 17:01:23.113086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.112891    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:01:23.113086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.112964    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:01:23.121426 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.121403    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:01:23.150928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.150905    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.151090 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.150978    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.151090 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.151029    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7m5\" (UniqueName: \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.151234 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.151102    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.151234 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:01:23.151134    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-serving-cert: secret "maas-api-e2e-worker-single-main-a7bea2-serving-cert" not found
Oct 31 17:01:23.151234 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:01:23.151200    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls podName:c9822d52-34c4-43f0-808d-cd235c6f5d28 nodeName:}" failed. No retries permitted until 2026-10-31 17:01:23.651177696 +0000 UTC m=+3528.712752872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls") pod "maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" (UID: "c9822d52-34c4-43f0-808d-cd235c6f5d28") : secret "maas-api-e2e-worker-single-main-a7bea2-serving-cert" not found
Oct 31 17:01:23.151854 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.151835    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.153505 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.153472    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.204509 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.204482    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7m5\" (UniqueName: \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.657664 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.657603    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.660000 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.659980    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:23.933674 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:23.933558    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:24.299747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:24.299717    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:01:24.306401 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:01:24.306371    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9822d52_34c4_43f0_808d_cd235c6f5d28.slice/crio-affd7cc19164d51065e60ffca51ebbb513867f6b8760e447841880c807c72406 WatchSource:0}: Error finding container affd7cc19164d51065e60ffca51ebbb513867f6b8760e447841880c807c72406: Status 404 returned error can't find the container with id affd7cc19164d51065e60ffca51ebbb513867f6b8760e447841880c807c72406
Oct 31 17:01:25.081701 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:25.081668    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" event={"ID":"c9822d52-34c4-43f0-808d-cd235c6f5d28","Type":"ContainerStarted","Data":"affd7cc19164d51065e60ffca51ebbb513867f6b8760e447841880c807c72406"}
Oct 31 17:01:26.087476 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:26.087439    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" event={"ID":"c9822d52-34c4-43f0-808d-cd235c6f5d28","Type":"ContainerStarted","Data":"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"}
Oct 31 17:01:26.087919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:26.087521    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:26.194328 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:26.194279    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" podStartSLOduration=3.268234014 podStartE2EDuration="4.194266587s" totalImagesPullingTime="926.032573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:01:26.193683327 +0000 UTC m=+3531.255258513" watchObservedRunningTime="2026-10-31 17:01:26.194266587 +0000 UTC m=+3531.255841767"
Oct 31 17:01:32.070009 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:32.069978    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:01:32.070598 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:32.070134    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:01:32.097832 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:32.097808    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:01:32.968681 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:01:32.968642    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:01:36.832300 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:36.832266    4417 scope.go:122] "RemoveContainer" containerID="0ce861628a18b2f9f70a932607c1086a16ecb81b20285e8ecf2cc0b6e61dd6ea"
Oct 31 17:01:36.842978 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:36.842957    4417 scope.go:122] "RemoveContainer" containerID="340e475e460812e1281aabdc57dd9bd5ea2c43f3dbfdd6383d15c3e1827bc6f8"
Oct 31 17:01:40.152406 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.152376    4417 generic.go:356] "Generic (PLEG): container finished" podID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerID="24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0" exitCode=0
Oct 31 17:01:40.152915 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.152420    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerDied","Data":"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"}
Oct 31 17:01:40.152915 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.152798    4417 scope.go:122] "RemoveContainer" containerID="24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"
Oct 31 17:01:40.153927 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.153897    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:40.154010 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.153966    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:40.154010 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:40.153996    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:01:41.166453 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:41.166418    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerStarted","Data":"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"}
Oct 31 17:01:41.693293 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:41.693254    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:41.694561 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:41.694536    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 17:01:41.694705 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:41.694582    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 17:01:42.693141 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:42.693107    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 17:01:42.693535 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:42.693161    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 17:01:43.693869 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:43.693832    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 17:01:43.694330 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:43.693892    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 17:01:44.027847 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:44.027764    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:44.693580 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:44.693549    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 17:01:44.693816 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:44.693602    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 17:01:45.041701 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.041030    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:01:45.064766 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.064731    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.103842 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.103802    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:01:45.158662 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.156872    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a7bea2\"/\"openshift-service-ca.crt\""
Oct 31 17:01:45.158662 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.156984    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a7bea2\"/\"simulator-single-main-a7bea2-kserve-self-signed-certs\""
Oct 31 17:01:45.158662 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.157104    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a7bea2\"/\"kube-root-ca.crt\""
Oct 31 17:01:45.159011 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.158857    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a7bea2\"/\"default-dockercfg-lv52t\""
Oct 31 17:01:45.200269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200225    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.200456 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200350    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.200456 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200425    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.200608 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200506    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjk2\" (UniqueName: \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.200608 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200534    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.200753 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.200676    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.301592 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301490    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.301592 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301542    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.301592 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301579    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.301592 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301595    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.302004 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301664    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjk2\" (UniqueName: \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.302004 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.301693    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.302130 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.302020    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.302130 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.302080    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.302130 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.302124    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.304001 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.303969    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.304117 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.304106    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.356333 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.356301    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjk2\" (UniqueName: \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\") pod \"simulator-single-main-a7bea2-kserve-555687b944-jpft4\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") " pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.376989 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.376964    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:45.693790 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.693711    4417 patch_prober.go:29] interesting pod/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 17:01:45.693790 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.693768    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 17:01:45.758827 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:45.749103    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:01:45.763933 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:01:45.763896    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e569fd_c3ce_4e6a_95cc_a40df6b70821.slice/crio-c8c77889d80b8e14dc313382c2eed7fb20d4a877d1d7d79ee652c57148c6208b WatchSource:0}: Error finding container c8c77889d80b8e14dc313382c2eed7fb20d4a877d1d7d79ee652c57148c6208b: Status 404 returned error can't find the container with id c8c77889d80b8e14dc313382c2eed7fb20d4a877d1d7d79ee652c57148c6208b
Oct 31 17:01:46.191753 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.191712    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" event={"ID":"82e569fd-c3ce-4e6a-95cc-a40df6b70821","Type":"ContainerStarted","Data":"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"}
Oct 31 17:01:46.191753 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.191758    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" event={"ID":"82e569fd-c3ce-4e6a-95cc-a40df6b70821","Type":"ContainerStarted","Data":"c8c77889d80b8e14dc313382c2eed7fb20d4a877d1d7d79ee652c57148c6208b"}
Oct 31 17:01:46.297697 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.297646    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" podStartSLOduration=1.297630915 podStartE2EDuration="1.297630915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:01:46.296495955 +0000 UTC m=+3551.358071140" watchObservedRunningTime="2026-10-31 17:01:46.297630915 +0000 UTC m=+3551.359206097"
Oct 31 17:01:46.377204 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.377175    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:46.697922 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.697891    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:46.698937 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:46.698916    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:01:55.377922 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:55.377884    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:55.392384 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:55.392347    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:01:56.247673 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:01:56.247625    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:02:02.943245 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:02:02.943212    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:02:32.948322 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:02:32.948283    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:02:36.878388 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:36.878348    4417 scope.go:122] "RemoveContainer" containerID="6243c251b9c1929ebaaa78b98d96d99b15a0be1dbbc1cc3de7b54a466484a7c2"
Oct 31 17:02:46.019255 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:46.019219    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:02:53.048455 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:53.048202    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:02:53.145643 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:53.145597    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:02:57.136218 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:57.136181    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:02:57.136688 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:57.136373    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" podUID="c9822d52-34c4-43f0-808d-cd235c6f5d28" containerName="maas-api" containerID="cri-o://18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9" gracePeriod=30
Oct 31 17:02:58.049551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.049509    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing" containerID="cri-o://e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554" gracePeriod=25
Oct 31 17:02:58.146928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.146888    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing" containerID="cri-o://31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924" gracePeriod=25
Oct 31 17:02:58.380109 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.380082    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:02:58.520592 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.520551    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") pod \"c9822d52-34c4-43f0-808d-cd235c6f5d28\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") "
Oct 31 17:02:58.520816 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.520675    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\" (UniqueName: \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\") pod \"c9822d52-34c4-43f0-808d-cd235c6f5d28\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") "
Oct 31 17:02:58.520816 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.520763    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\") pod \"c9822d52-34c4-43f0-808d-cd235c6f5d28\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") "
Oct 31 17:02:58.520968 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.520816    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\") pod \"c9822d52-34c4-43f0-808d-cd235c6f5d28\" (UID: \"c9822d52-34c4-43f0-808d-cd235c6f5d28\") "
Oct 31 17:02:58.521196 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.521166    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca" pod "c9822d52-34c4-43f0-808d-cd235c6f5d28" (UID: "c9822d52-34c4-43f0-808d-cd235c6f5d28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:02:58.523143 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.523120    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5" pod "c9822d52-34c4-43f0-808d-cd235c6f5d28" (UID: "c9822d52-34c4-43f0-808d-cd235c6f5d28"). InnerVolumeSpecName "kube-api-access-kv7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:02:58.523344 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.523319    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs" pod "c9822d52-34c4-43f0-808d-cd235c6f5d28" (UID: "c9822d52-34c4-43f0-808d-cd235c6f5d28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:02:58.523431 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.523375    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls" pod "c9822d52-34c4-43f0-808d-cd235c6f5d28" (UID: "c9822d52-34c4-43f0-808d-cd235c6f5d28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:02:58.536939 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.536911    4417 generic.go:356] "Generic (PLEG): container finished" podID="c9822d52-34c4-43f0-808d-cd235c6f5d28" containerID="18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9" exitCode=0
Oct 31 17:02:58.537052 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.536954    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"
Oct 31 17:02:58.537052 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.536984    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" event={"ID":"c9822d52-34c4-43f0-808d-cd235c6f5d28","Type":"ContainerDied","Data":"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"}
Oct 31 17:02:58.537052 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.537044    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc" event={"ID":"c9822d52-34c4-43f0-808d-cd235c6f5d28","Type":"ContainerDied","Data":"affd7cc19164d51065e60ffca51ebbb513867f6b8760e447841880c807c72406"}
Oct 31 17:02:58.537170 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.537074    4417 scope.go:122] "RemoveContainer" containerID="18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"
Oct 31 17:02:58.546981 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.546962    4417 scope.go:122] "RemoveContainer" containerID="18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"
Oct 31 17:02:58.547239 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:02:58.547219    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9\": container with ID starting with 18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9 not found: ID does not exist" containerID="18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"
Oct 31 17:02:58.547294 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.547249    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9"} err="failed to get container status \"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9\": rpc error: code = NotFound desc = could not find container \"18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9\": container with ID starting with 18f1efde7f8bbf7d7ff67f1f9629b3dca5cd0a2aeca33a5f14e0c1bd22697ce9 not found: ID does not exist"
Oct 31 17:02:58.622751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.622669    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9822d52-34c4-43f0-808d-cd235c6f5d28-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:58.622751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.622699    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:58.622751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.622711    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9822d52-34c4-43f0-808d-cd235c6f5d28-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:58.622751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.622721    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7m5\" (UniqueName: \"kubernetes.io/projected/c9822d52-34c4-43f0-808d-cd235c6f5d28-kube-api-access-kv7m5\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:02:58.687164 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.687133    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:02:58.690359 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:58.690331    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a7bea2-868f47c684-xnwwc"]
Oct 31 17:02:59.049989 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:02:59.049947    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9822d52-34c4-43f0-808d-cd235c6f5d28" path="/var/lib/kubelet/pods/c9822d52-34c4-43f0-808d-cd235c6f5d28/volumes"
Oct 31 17:03:01.039894 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.039852    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerName="main" containerID="cri-o://06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae" gracePeriod=45
Oct 31 17:03:01.295680 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.295598    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:03:01.451230 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451194    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451243    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451281    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\" (UniqueName: \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451366    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451411    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\" (UniqueName: \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451455    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\") pod \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\" (UID: \"82e569fd-c3ce-4e6a-95cc-a40df6b70821\") "
Oct 31 17:03:01.451826 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451580    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:01.451875 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451814    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:01.451875 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.451828    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:01.452177 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.452095    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.452177 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.452141    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.452177 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.452155    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.453818 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.453791    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:01.453943 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.453897    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:03:01.454235 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.454214    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2" pod "82e569fd-c3ce-4e6a-95cc-a40df6b70821" (UID: "82e569fd-c3ce-4e6a-95cc-a40df6b70821"). InnerVolumeSpecName "kube-api-access-nnjk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:03:01.553386 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.553293    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e569fd-c3ce-4e6a-95cc-a40df6b70821-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.553386 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.553319    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjk2\" (UniqueName: \"kubernetes.io/projected/82e569fd-c3ce-4e6a-95cc-a40df6b70821-kube-api-access-nnjk2\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.553386 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.553331    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e569fd-c3ce-4e6a-95cc-a40df6b70821-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:01.554357 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.554332    4417 generic.go:356] "Generic (PLEG): container finished" podID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerID="06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae" exitCode=0
Oct 31 17:03:01.554421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.554373    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"
Oct 31 17:03:01.554421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.554390    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" event={"ID":"82e569fd-c3ce-4e6a-95cc-a40df6b70821","Type":"ContainerDied","Data":"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"}
Oct 31 17:03:01.554489 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.554434    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" event={"ID":"82e569fd-c3ce-4e6a-95cc-a40df6b70821","Type":"ContainerDied","Data":"c8c77889d80b8e14dc313382c2eed7fb20d4a877d1d7d79ee652c57148c6208b"}
Oct 31 17:03:01.554489 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.554453    4417 scope.go:122] "RemoveContainer" containerID="06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"
Oct 31 17:03:01.564751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.564730    4417 scope.go:122] "RemoveContainer" containerID="06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"
Oct 31 17:03:01.564990 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:01.564971    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae\": container with ID starting with 06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae not found: ID does not exist" containerID="06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"
Oct 31 17:03:01.565048 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.565001    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae"} err="failed to get container status \"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae\": rpc error: code = NotFound desc = could not find container \"06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae\": container with ID starting with 06976ce59bde6b2d3b4faf680a6c79ceeb1a8b818eb4e056d9d4fb3ce38347ae not found: ID does not exist"
Oct 31 17:03:01.703885 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.703856    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:03:01.709076 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:01.709051    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4"]
Oct 31 17:03:02.068525 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:02.068482    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 17:03:02.068939 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:02.068546    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 17:03:02.237696 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:02.237646    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-a7bea2/simulator-single-main-a7bea2-kserve-555687b944-jpft4" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:03:02.943398 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:02.943351    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:03:03.049038 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:03.049007    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" path="/var/lib/kubelet/pods/82e569fd-c3ce-4e6a-95cc-a40df6b70821/volumes"
Oct 31 17:03:06.147008 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:06.146977    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:03:06.147414 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:06.147175    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy" containerID="cri-o://851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340" gracePeriod=30
Oct 31 17:03:11.403217 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.403190    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:03:11.430383 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430308    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430383 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430348    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430594 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430394    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430594 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430438    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430743 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430660    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430743 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430725    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430743 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430732    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:11.430919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430763    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430821    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.430919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430848    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:03:11.430919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.430877    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\") pod \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\" (UID: \"cc02ffcd-8ddd-4093-8d0e-779161a7c787\") "
Oct 31 17:03:11.431151 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431042    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:11.431151 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431120    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:11.431291 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431267    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:11.431352 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431275    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.431352 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431315    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.431352 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431331    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.431352 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.431346    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.432775 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.432753    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "kube-api-access-6fcfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:03:11.432775 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.432767    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:03:11.432932 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.432919    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 17:03:11.432992 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.432928    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy" pod "cc02ffcd-8ddd-4093-8d0e-779161a7c787" (UID: "cc02ffcd-8ddd-4093-8d0e-779161a7c787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:03:11.532441 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.532411    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.532441 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.532439    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcfs\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-kube-api-access-6fcfs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.532677 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.532449    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.532677 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.532459    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.532677 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.532467    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02ffcd-8ddd-4093-8d0e-779161a7c787-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:11.602140 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.602106    4417 generic.go:356] "Generic (PLEG): container finished" podID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerID="851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340" exitCode=0
Oct 31 17:03:11.602296 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.602167    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"
Oct 31 17:03:11.602296 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.602185    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerDied","Data":"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"}
Oct 31 17:03:11.602296 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.602239    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk" event={"ID":"cc02ffcd-8ddd-4093-8d0e-779161a7c787","Type":"ContainerDied","Data":"0351a7dfe37eabae343fffe50913a6d45f24c38b3cbfacbd1abe4abe2247df6f"}
Oct 31 17:03:11.602296 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.602269    4417 scope.go:122] "RemoveContainer" containerID="851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"
Oct 31 17:03:11.611451 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.611428    4417 scope.go:122] "RemoveContainer" containerID="24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"
Oct 31 17:03:11.619162 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.619143    4417 scope.go:122] "RemoveContainer" containerID="851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"
Oct 31 17:03:11.619408 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:11.619387    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340\": container with ID starting with 851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340 not found: ID does not exist" containerID="851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"
Oct 31 17:03:11.619461 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.619418    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340"} err="failed to get container status \"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340\": rpc error: code = NotFound desc = could not find container \"851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340\": container with ID starting with 851c848d128fcbe98e01a564c7be96e8047c04103e59d7738a3ae04fc7239340 not found: ID does not exist"
Oct 31 17:03:11.619461 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.619436    4417 scope.go:122] "RemoveContainer" containerID="24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"
Oct 31 17:03:11.619711 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:11.619695    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0\": container with ID starting with 24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0 not found: ID does not exist" containerID="24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"
Oct 31 17:03:11.619750 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.619719    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0"} err="failed to get container status \"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0\": rpc error: code = NotFound desc = could not find container \"24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0\": container with ID starting with 24c6f85110b841d28b2378140ec361b381cb7df1d6d9425e23e5621b337351f0 not found: ID does not exist"
Oct 31 17:03:11.753088 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.753001    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:03:11.757938 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:11.757911    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a7bea2-openshift-default-58b6cb87c-nf7lk"]
Oct 31 17:03:12.067972 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:12.067930    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 17:03:12.068144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:12.067931    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 17:03:13.048107 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.048069    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" path="/var/lib/kubelet/pods/cc02ffcd-8ddd-4093-8d0e-779161a7c787/volumes"
Oct 31 17:03:13.563686 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.563656    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:03:13.564034 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564023    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:13.564080 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564035    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:13.564080 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564049    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c9822d52-34c4-43f0-808d-cd235c6f5d28" containerName="maas-api"
Oct 31 17:03:13.564080 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564055    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9822d52-34c4-43f0-808d-cd235c6f5d28" containerName="maas-api"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564080    4417 cpu_manager.go:414] "containerMap: removing container" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerName="main"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564086    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerName="main"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564136    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9822d52-34c4-43f0-808d-cd235c6f5d28" containerName="maas-api"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564146    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564154    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e569fd-c3ce-4e6a-95cc-a40df6b70821" containerName="main"
Oct 31 17:03:13.564182 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.564162    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:13.568891 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.568870    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653394    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653590 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653443    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653590 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653553    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653590 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653584    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653729 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653604    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncmw\" (UniqueName: \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.653770 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.653725    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.662814 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.662792    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5fe21cf8-kserve-self-signed-certs\""
Oct 31 17:03:13.664628 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.664601    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:03:13.754993 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.754963    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755011    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755034    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755062    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncmw\" (UniqueName: \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755098    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755403 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755178    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755656 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755629    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755786 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755603    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.755786 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.755758    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.757428 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.757408    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.757765 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.757746    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.807493 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.807455    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncmw\" (UniqueName: \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\") pod \"e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") " pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:13.880848 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:13.880775    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:14.256785 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:14.256708    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:03:14.263435 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:03:14.263407    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc4b351_3927_4826_8aca_4f2606a77c0b.slice/crio-ac035424800374b3cd9615c9fc44b265fb2b2b5e1e5b43aef8a98c5d40582512 WatchSource:0}: Error finding container ac035424800374b3cd9615c9fc44b265fb2b2b5e1e5b43aef8a98c5d40582512: Status 404 returned error can't find the container with id ac035424800374b3cd9615c9fc44b265fb2b2b5e1e5b43aef8a98c5d40582512
Oct 31 17:03:14.618035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:14.617998    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" event={"ID":"bbc4b351-3927-4826-8aca-4f2606a77c0b","Type":"ContainerStarted","Data":"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"}
Oct 31 17:03:14.618228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:14.618049    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" event={"ID":"bbc4b351-3927-4826-8aca-4f2606a77c0b","Type":"ContainerStarted","Data":"ac035424800374b3cd9615c9fc44b265fb2b2b5e1e5b43aef8a98c5d40582512"}
Oct 31 17:03:14.725422 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:14.725365    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" podStartSLOduration=1.725351592 podStartE2EDuration="1.725351592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:03:14.724796999 +0000 UTC m=+3639.786372188" watchObservedRunningTime="2026-10-31 17:03:14.725351592 +0000 UTC m=+3639.786926772"
Oct 31 17:03:14.881277 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:14.881180    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:18.319349 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.319281    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:03:18.403664 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.402009    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\") pod \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") "
Oct 31 17:03:18.403664 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.402114    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\" (UniqueName: \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\") pod \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\" (UID: \"145ca58b-ed57-49d2-ab0b-760e4a7cc28e\") "
Oct 31 17:03:18.403664 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.402663    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume" pod "145ca58b-ed57-49d2-ab0b-760e4a7cc28e" (UID: "145ca58b-ed57-49d2-ab0b-760e4a7cc28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:03:18.406648 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.405775    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm" pod "145ca58b-ed57-49d2-ab0b-760e4a7cc28e" (UID: "145ca58b-ed57-49d2-ab0b-760e4a7cc28e"). InnerVolumeSpecName "kube-api-access-dnhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:03:18.431794 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.431766    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:03:18.503313 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.503272    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\" (UniqueName: \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\") pod \"d4271107-57c8-4b2a-879a-aa282b84b492\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") "
Oct 31 17:03:18.503516 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.503460    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\") pod \"d4271107-57c8-4b2a-879a-aa282b84b492\" (UID: \"d4271107-57c8-4b2a-879a-aa282b84b492\") "
Oct 31 17:03:18.503803 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.503776    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:18.503902 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.503804    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhjm\" (UniqueName: \"kubernetes.io/projected/145ca58b-ed57-49d2-ab0b-760e4a7cc28e-kube-api-access-dnhjm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:18.503992 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.503963    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume" pod "d4271107-57c8-4b2a-879a-aa282b84b492" (UID: "d4271107-57c8-4b2a-879a-aa282b84b492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:03:18.505702 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.505671    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv" pod "d4271107-57c8-4b2a-879a-aa282b84b492" (UID: "d4271107-57c8-4b2a-879a-aa282b84b492"). InnerVolumeSpecName "kube-api-access-cn5hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:03:18.605235 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.605123    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4271107-57c8-4b2a-879a-aa282b84b492-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:18.605235 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.605161    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn5hv\" (UniqueName: \"kubernetes.io/projected/d4271107-57c8-4b2a-879a-aa282b84b492-kube-api-access-cn5hv\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:03:18.639194 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.639151    4417 generic.go:356] "Generic (PLEG): container finished" podID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerID="e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554" exitCode=0
Oct 31 17:03:18.639400 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.639293    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" event={"ID":"145ca58b-ed57-49d2-ab0b-760e4a7cc28e","Type":"ContainerDied","Data":"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"}
Oct 31 17:03:18.639400 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.639354    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf" event={"ID":"145ca58b-ed57-49d2-ab0b-760e4a7cc28e","Type":"ContainerDied","Data":"b288be9672332e4bd1093c058cb77373fa748eef23dd5f9464b8a9fb9f150e70"}
Oct 31 17:03:18.639400 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.639386    4417 scope.go:122] "RemoveContainer" containerID="e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"
Oct 31 17:03:18.639600 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.639533    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"
Oct 31 17:03:18.644496 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.644464    4417 generic.go:356] "Generic (PLEG): container finished" podID="d4271107-57c8-4b2a-879a-aa282b84b492" containerID="31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924" exitCode=0
Oct 31 17:03:18.644744 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.644550    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" event={"ID":"d4271107-57c8-4b2a-879a-aa282b84b492","Type":"ContainerDied","Data":"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"}
Oct 31 17:03:18.644744 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.644598    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t" event={"ID":"d4271107-57c8-4b2a-879a-aa282b84b492","Type":"ContainerDied","Data":"44c05bafa40764acfc8075717a34ca8e641c026cad6f8f63b23e551876928e4c"}
Oct 31 17:03:18.644744 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.644692    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"
Oct 31 17:03:18.657958 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.657883    4417 scope.go:122] "RemoveContainer" containerID="e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"
Oct 31 17:03:18.658408 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:18.658363    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554\": container with ID starting with e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554 not found: ID does not exist" containerID="e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"
Oct 31 17:03:18.658523 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.658417    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554"} err="failed to get container status \"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554\": rpc error: code = NotFound desc = could not find container \"e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554\": container with ID starting with e468194cc0b461afa47447dc0ea5af980a6a3e3518b3dcc16e20e8219cb72554 not found: ID does not exist"
Oct 31 17:03:18.658523 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.658442    4417 scope.go:122] "RemoveContainer" containerID="31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"
Oct 31 17:03:18.674525 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.674434    4417 scope.go:122] "RemoveContainer" containerID="31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"
Oct 31 17:03:18.676900 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:18.674930    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924\": container with ID starting with 31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924 not found: ID does not exist" containerID="31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"
Oct 31 17:03:18.676900 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.674965    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924"} err="failed to get container status \"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924\": rpc error: code = NotFound desc = could not find container \"31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924\": container with ID starting with 31b996f40e1d8fff40747756a25db847ae5820799245bff18c6669c4f8814924 not found: ID does not exist"
Oct 31 17:03:18.793904 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.793851    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:03:18.797119 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.797086    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a7bea2-658b44bc-vcltf"]
Oct 31 17:03:18.897069 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.896971    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:03:18.900942 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:18.900909    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a7bea2-dc8877xs6t"]
Oct 31 17:03:19.076165 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:19.076065    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" path="/var/lib/kubelet/pods/145ca58b-ed57-49d2-ab0b-760e4a7cc28e/volumes"
Oct 31 17:03:19.077015 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:19.076987    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" path="/var/lib/kubelet/pods/d4271107-57c8-4b2a-879a-aa282b84b492/volumes"
Oct 31 17:03:23.881511 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:23.881475    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:23.894291 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:23.894263    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:24.719443 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:24.719413    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:03:29.049054 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:29.049021    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"]
Oct 31 17:03:29.086634 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:29.086588    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wcr4b"]
Oct 31 17:03:31.047399 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:31.047366    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a991b4-913c-4f4e-b14a-69bff5034e5a" path="/var/lib/kubelet/pods/53a991b4-913c-4f4e-b14a-69bff5034e5a/volumes"
Oct 31 17:03:32.942753 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:03:32.942715    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:03:53.740090 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740049    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:03:53.740641 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740592    4417 cpu_manager.go:414] "containerMap: removing container" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:53.740641 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740631    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc02ffcd-8ddd-4093-8d0e-779161a7c787" containerName="istio-proxy"
Oct 31 17:03:53.740805 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740663    4417 cpu_manager.go:414] "containerMap: removing container" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing"
Oct 31 17:03:53.740805 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740671    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing"
Oct 31 17:03:53.740805 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740682    4417 cpu_manager.go:414] "containerMap: removing container" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing"
Oct 31 17:03:53.740805 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740689    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing"
Oct 31 17:03:53.741021 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740836    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4271107-57c8-4b2a-879a-aa282b84b492" containerName="payload-pre-processing"
Oct 31 17:03:53.741021 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.740848    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="145ca58b-ed57-49d2-ab0b-760e4a7cc28e" containerName="payload-processing"
Oct 31 17:03:53.744225 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.744201    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.778938 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.778907    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.779068 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.778959    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.779068 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.778988    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.779172 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.779106    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpjc\" (UniqueName: \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.779210 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.779184    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.779246 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.779235    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.814555 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.814527    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5fe21cf8-kserve-self-signed-certs\""
Oct 31 17:03:53.816454 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.816432    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:03:53.880252 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880224    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.880430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880261    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.880430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880302    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpjc\" (UniqueName: \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.880430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880372    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.881160 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880439    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.881160 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880464    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.881160 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880776    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.881160 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880845    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.881160 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.880864    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.883115 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.883080    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.883722 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.883676    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:53.933293 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:53.933262    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpjc\" (UniqueName: \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\") pod \"e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") " pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:54.055737 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:54.055707    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:03:54.431674 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:54.431569    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:03:54.438962 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:03:54.438931    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f8b8256_30c5_4d8f_af7a_a3237106d162.slice/crio-434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083 WatchSource:0}: Error finding container 434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083: Status 404 returned error can't find the container with id 434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083
Oct 31 17:03:54.843109 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:54.843068    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" event={"ID":"0f8b8256-30c5-4d8f-af7a-a3237106d162","Type":"ContainerStarted","Data":"5377466b50a3486e8587db356030ab11f658cc28f0ab99396f61a18521080b63"}
Oct 31 17:03:54.843109 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:54.843113    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" event={"ID":"0f8b8256-30c5-4d8f-af7a-a3237106d162","Type":"ContainerStarted","Data":"434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083"}
Oct 31 17:03:54.948957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:54.948902    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" podStartSLOduration=1.948889066 podStartE2EDuration="1.948889066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:03:54.947483987 +0000 UTC m=+3680.009059173" watchObservedRunningTime="2026-10-31 17:03:54.948889066 +0000 UTC m=+3680.010464251"
Oct 31 17:03:55.055792 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:03:55.055758    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:02.296798 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:02.296748    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:04:02.297246 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:02.296815    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:04:02.943735 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:04:02.943695    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:04:04.056163 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:04.056120    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:04.074669 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:04.074638    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:04.925269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:04.925236    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:24.642462 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.642425    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"]
Oct 31 17:04:24.646141 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.646110    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.694463 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.694431    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:04:24.694463 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.694437    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:04:24.734633 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.734595    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.734830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.734777    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.734830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.734811    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qdk\" (UniqueName: \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.740221 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.740196    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:04:24.740221 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.740227    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:04:24.742311 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.742287    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"]
Oct 31 17:04:24.835333 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.835300    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.835333 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.835339    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qdk\" (UniqueName: \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.835567 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.835386    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.836148 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.836122    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.837776 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.837747    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.888121 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.888088    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qdk\" (UniqueName: \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\") pod \"maas-api-revoke-sub-452b6d7b-669wz\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:24.959526 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:24.959425    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:25.330087 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:25.330052    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"]
Oct 31 17:04:25.338342 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:04:25.338313    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb8a1a2_5c72_4b29_a126_8dfb84705795.slice/crio-74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929 WatchSource:0}: Error finding container 74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929: Status 404 returned error can't find the container with id 74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929
Oct 31 17:04:26.007078 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:26.006982    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz" event={"ID":"efb8a1a2-5c72-4b29-a126-8dfb84705795","Type":"ContainerStarted","Data":"e929efdbb5d1c906c5ca11572d3cfaedb9e80f2e70067c16bb55b8fd9ef91d34"}
Oct 31 17:04:26.007078 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:26.007042    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz" event={"ID":"efb8a1a2-5c72-4b29-a126-8dfb84705795","Type":"ContainerStarted","Data":"74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929"}
Oct 31 17:04:26.111371 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:26.111318    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz" podStartSLOduration=2.111305502 podStartE2EDuration="2.111305502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:04:26.110147679 +0000 UTC m=+3711.171722868" watchObservedRunningTime="2026-10-31 17:04:26.111305502 +0000 UTC m=+3711.172880688"
Oct 31 17:04:27.015594 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:27.015552    4417 generic.go:356] "Generic (PLEG): container finished" podID="efb8a1a2-5c72-4b29-a126-8dfb84705795" containerID="e929efdbb5d1c906c5ca11572d3cfaedb9e80f2e70067c16bb55b8fd9ef91d34" exitCode=0
Oct 31 17:04:27.016149 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:27.015649    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz" event={"ID":"efb8a1a2-5c72-4b29-a126-8dfb84705795","Type":"ContainerDied","Data":"e929efdbb5d1c906c5ca11572d3cfaedb9e80f2e70067c16bb55b8fd9ef91d34"}
Oct 31 17:04:28.406702 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:28.405846    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8965v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:04:28.406702 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:28.405916    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8965v" podUID="21201683-d4a2-4c03-a84f-ff1803240663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:04:28.989827 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:28.972865    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:29.076639 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.076079    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\" (UniqueName: \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\") pod \"efb8a1a2-5c72-4b29-a126-8dfb84705795\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") "
Oct 31 17:04:29.076639 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.076159    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\") pod \"efb8a1a2-5c72-4b29-a126-8dfb84705795\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") "
Oct 31 17:04:29.076639 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.076225    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\") pod \"efb8a1a2-5c72-4b29-a126-8dfb84705795\" (UID: \"efb8a1a2-5c72-4b29-a126-8dfb84705795\") "
Oct 31 17:04:29.082634 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.077050    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca" pod "efb8a1a2-5c72-4b29-a126-8dfb84705795" (UID: "efb8a1a2-5c72-4b29-a126-8dfb84705795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:04:29.095860 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.095826    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk" pod "efb8a1a2-5c72-4b29-a126-8dfb84705795" (UID: "efb8a1a2-5c72-4b29-a126-8dfb84705795"). InnerVolumeSpecName "kube-api-access-x9qdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:04:29.134135 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.134098    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"
Oct 31 17:04:29.141231 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.135742    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca" pod "efb8a1a2-5c72-4b29-a126-8dfb84705795" (UID: "efb8a1a2-5c72-4b29-a126-8dfb84705795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:04:29.185648 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.181360    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:29.185648 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.181406    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb8a1a2-5c72-4b29-a126-8dfb84705795-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:29.185648 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.181422    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qdk\" (UniqueName: \"kubernetes.io/projected/efb8a1a2-5c72-4b29-a126-8dfb84705795-kube-api-access-x9qdk\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:29.270304 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.270210    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz" event={"ID":"efb8a1a2-5c72-4b29-a126-8dfb84705795","Type":"ContainerDied","Data":"74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929"}
Oct 31 17:04:29.270567 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:29.270544    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b95ef0ae4ff3629a45d2ac0e40f673c5f832e0adc60b3238ac6bce0ee55929"
Oct 31 17:04:32.944035 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:04:32.943999    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:04:36.922745 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:36.922710    4417 scope.go:122] "RemoveContainer" containerID="a873b27983507bad436ec1e93eb19e5035a085334ba2f5bbb28aefc4c6145129"
Oct 31 17:04:37.560594 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:37.560545    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:04:42.131701 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:42.131667    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:04:46.278779 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.278742    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"]
Oct 31 17:04:46.279303 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.279281    4417 cpu_manager.go:414] "containerMap: removing container" podUID="efb8a1a2-5c72-4b29-a126-8dfb84705795" containerName="revoke-keys"
Oct 31 17:04:46.279385 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.279302    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb8a1a2-5c72-4b29-a126-8dfb84705795" containerName="revoke-keys"
Oct 31 17:04:46.279465 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.279448    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb8a1a2-5c72-4b29-a126-8dfb84705795" containerName="revoke-keys"
Oct 31 17:04:46.282566 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.282537    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.330905 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.330872    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:04:46.331077 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.330883    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:04:46.377065 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.377028    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:04:46.377251 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.377090    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:04:46.380222 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.380198    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"]
Oct 31 17:04:46.417627 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.417583    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bmk\" (UniqueName: \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.417780 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.417634    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.417780 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.417750    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.518409 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.518375    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.518576 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.518466    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bmk\" (UniqueName: \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.518576 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.518488    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.519122 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.519099    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.520722 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.520704    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.571122 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.571088    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bmk\" (UniqueName: \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\") pod \"maas-api-revoke-sub-3e31a506-fhzlz\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.600803 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.600779    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:46.967231 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:46.967124    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"]
Oct 31 17:04:46.974890 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:04:46.974849    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fedf30_bdc6_4e2a_8e2c_aa02aa988a02.slice/crio-a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969 WatchSource:0}: Error finding container a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969: Status 404 returned error can't find the container with id a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969
Oct 31 17:04:47.242477 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:47.242450    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz" event={"ID":"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02","Type":"ContainerStarted","Data":"a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969"}
Oct 31 17:04:48.251804 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:48.251766    4417 generic.go:356] "Generic (PLEG): container finished" podID="a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" containerID="398d19122301e29c89758295f994b549229891791265b0f71b86277853cff263" exitCode=0
Oct 31 17:04:48.252320 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:48.251822    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz" event={"ID":"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02","Type":"ContainerDied","Data":"398d19122301e29c89758295f994b549229891791265b0f71b86277853cff263"}
Oct 31 17:04:50.018365 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.018333    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:50.103273 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.103242    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\" (UniqueName: \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\") pod \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") "
Oct 31 17:04:50.103577 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.103547    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\") pod \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") "
Oct 31 17:04:50.103931 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.103900    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\") pod \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\" (UID: \"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02\") "
Oct 31 17:04:50.114656 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.111019    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk" pod "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" (UID: "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02"). InnerVolumeSpecName "kube-api-access-54bmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:04:50.114656 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.113807    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca" pod "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" (UID: "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:04:50.135649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.123054    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca" pod "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" (UID: "a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:04:50.237581 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.209250    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:50.237581 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.209296    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bmk\" (UniqueName: \"kubernetes.io/projected/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-kube-api-access-54bmk\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:50.237581 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.209346    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:50.370266 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.370222    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz" event={"ID":"a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02","Type":"ContainerDied","Data":"a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969"}
Oct 31 17:04:50.375071 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.375047    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b3709b246fc7e2381a2a2ad0b5f630e6143d0cf163d15ac97e39f424691969"
Oct 31 17:04:50.375284 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:50.374779    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"
Oct 31 17:04:52.580033 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.579996    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerName="main" containerID="cri-o://6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54" gracePeriod=45
Oct 31 17:04:52.709086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.709050    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.210:8000/ready\": dial tcp 10.133.0.210:8000: connect: connection refused"
Oct 31 17:04:52.825984 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.825961    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:04:52.849076 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849012    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849193 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849103    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849193 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849139    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849193 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849168    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849370 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849201    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849370 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849256    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:52.849370 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849309    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\" (UniqueName: \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\") pod \"bbc4b351-3927-4826-8aca-4f2606a77c0b\" (UID: \"bbc4b351-3927-4826-8aca-4f2606a77c0b\") "
Oct 31 17:04:52.849534 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849414    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:52.849534 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849431    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:52.849718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849689    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:52.849811 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849720    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:52.849811 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.849735    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:52.851407 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.851385    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:04:52.851485 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.851445    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:52.851485 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.851443    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw" pod "bbc4b351-3927-4826-8aca-4f2606a77c0b" (UID: "bbc4b351-3927-4826-8aca-4f2606a77c0b"). InnerVolumeSpecName "kube-api-access-kncmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:04:52.950957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.950932    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc4b351-3927-4826-8aca-4f2606a77c0b-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:52.950957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.950956    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc4b351-3927-4826-8aca-4f2606a77c0b-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:52.951129 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:52.950966    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kncmw\" (UniqueName: \"kubernetes.io/projected/bbc4b351-3927-4826-8aca-4f2606a77c0b-kube-api-access-kncmw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:53.386338 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.386304    4417 generic.go:356] "Generic (PLEG): container finished" podID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerID="6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54" exitCode=0
Oct 31 17:04:53.386551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.386343    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"
Oct 31 17:04:53.386551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.386387    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" event={"ID":"bbc4b351-3927-4826-8aca-4f2606a77c0b","Type":"ContainerDied","Data":"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"}
Oct 31 17:04:53.386551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.386429    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq" event={"ID":"bbc4b351-3927-4826-8aca-4f2606a77c0b","Type":"ContainerDied","Data":"ac035424800374b3cd9615c9fc44b265fb2b2b5e1e5b43aef8a98c5d40582512"}
Oct 31 17:04:53.386551 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.386447    4417 scope.go:122] "RemoveContainer" containerID="6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"
Oct 31 17:04:53.395010 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.394984    4417 scope.go:122] "RemoveContainer" containerID="6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"
Oct 31 17:04:53.395267 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:04:53.395247    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54\": container with ID starting with 6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54 not found: ID does not exist" containerID="6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"
Oct 31 17:04:53.395328 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.395279    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54"} err="failed to get container status \"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54\": rpc error: code = NotFound desc = could not find container \"6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54\": container with ID starting with 6b48de6ef1d8f77e275f2c03c4a537ccc296e69ad64829500a2e31dd69ac2f54 not found: ID does not exist"
Oct 31 17:04:53.537385 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.537347    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:04:53.583334 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:53.583304    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5fe21cf8-kserve-56cbc8599c-rtkkq"]
Oct 31 17:04:55.048898 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:55.048860    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" path="/var/lib/kubelet/pods/bbc4b351-3927-4826-8aca-4f2606a77c0b/volumes"
Oct 31 17:04:57.168376 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.168336    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" podUID="0f8b8256-30c5-4d8f-af7a-a3237106d162" containerName="main" containerID="cri-o://5377466b50a3486e8587db356030ab11f658cc28f0ab99396f61a18521080b63" gracePeriod=45
Oct 31 17:04:57.405143 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.405116    4417 generic.go:356] "Generic (PLEG): container finished" podID="0f8b8256-30c5-4d8f-af7a-a3237106d162" containerID="5377466b50a3486e8587db356030ab11f658cc28f0ab99396f61a18521080b63" exitCode=0
Oct 31 17:04:57.405279 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.405196    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" event={"ID":"0f8b8256-30c5-4d8f-af7a-a3237106d162","Type":"ContainerDied","Data":"5377466b50a3486e8587db356030ab11f658cc28f0ab99396f61a18521080b63"}
Oct 31 17:04:57.405279 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.405248    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz" event={"ID":"0f8b8256-30c5-4d8f-af7a-a3237106d162","Type":"ContainerDied","Data":"434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083"}
Oct 31 17:04:57.405279 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.405266    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434c243fcdb2d6a6458d554f547e4137321dd637b79d418a16e3d1801d958083"
Oct 31 17:04:57.416652 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.416609    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:57.496639 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496545    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496639 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496578    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496834 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496656    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496834 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496719    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\" (UniqueName: \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496834 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496779    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496997 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496829    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\") pod \"0f8b8256-30c5-4d8f-af7a-a3237106d162\" (UID: \"0f8b8256-30c5-4d8f-af7a-a3237106d162\") "
Oct 31 17:04:57.496997 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.496922    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:57.497125 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.497100    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:57.497125 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.497117    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:57.497228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.497183    4417 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-tmp-dir\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:57.498809 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.498794    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:04:57.499225 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.499208    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "kube-api-access-2cpjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:04:57.499225 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.499211    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs" pod "0f8b8256-30c5-4d8f-af7a-a3237106d162" (UID: "0f8b8256-30c5-4d8f-af7a-a3237106d162"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:04:57.598234 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.598196    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cpjc\" (UniqueName: \"kubernetes.io/projected/0f8b8256-30c5-4d8f-af7a-a3237106d162-kube-api-access-2cpjc\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:57.598234 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.598228    4417 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-model-cache\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:57.598234 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.598237    4417 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-home\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:57.598478 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.598248    4417 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f8b8256-30c5-4d8f-af7a-a3237106d162-dshm\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:57.598478 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:57.598256    4417 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f8b8256-30c5-4d8f-af7a-a3237106d162-tls-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:04:58.409633 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:58.409586    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"
Oct 31 17:04:58.560126 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:58.560096    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:04:58.606248 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:58.606212    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5fe21cf8-kserve-648987476f-5s9gz"]
Oct 31 17:04:59.049367 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:04:59.049327    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8b8256-30c5-4d8f-af7a-a3237106d162" path="/var/lib/kubelet/pods/0f8b8256-30c5-4d8f-af7a-a3237106d162/volumes"
Oct 31 17:05:02.306820 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:02.306756    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-x6tl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:05:02.307525 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:02.307490    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-x6tl6" podUID="ba404bf8-c02c-4e49-b4a8-41968ddb20b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:05:02.334861 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:02.334818    4417 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-s8bll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 17:05:02.335192 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:02.335146    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-s8bll" podUID="312a05f7-0e52-4001-88c9-b40cc107b30a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 17:05:02.964231 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:05:02.964141    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:05:19.036196 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036166    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"]
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036530    4417 cpu_manager.go:414] "containerMap: removing container" podUID="0f8b8256-30c5-4d8f-af7a-a3237106d162" containerName="main"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036540    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8b8256-30c5-4d8f-af7a-a3237106d162" containerName="main"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036558    4417 cpu_manager.go:414] "containerMap: removing container" podUID="a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" containerName="revoke-keys"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036563    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" containerName="revoke-keys"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036586    4417 cpu_manager.go:414] "containerMap: removing container" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerName="main"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036591    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerName="main"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036667    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8b8256-30c5-4d8f-af7a-a3237106d162" containerName="main"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036679    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" containerName="revoke-keys"
Oct 31 17:05:19.036830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.036689    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc4b351-3927-4826-8aca-4f2606a77c0b" containerName="main"
Oct 31 17:05:19.039894 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.039871    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.088066 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.088033    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 17:05:19.134138 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.134102    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:05:19.134138 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.134102    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:05:19.134342 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.134127    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\""
Oct 31 17:05:19.136088 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.136057    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"]
Oct 31 17:05:19.169323 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.169296    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.169485 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.169355    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hznv\" (UniqueName: \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.169485 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.169453    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.270266 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.270232    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.270439 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.270352    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.270439 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.270422    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hznv\" (UniqueName: \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.270910 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.270892    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.272650 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.272632    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.322792 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.322765    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hznv\" (UniqueName: \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\") pod \"maas-api-revoke-sub-317514eb-njgfg\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.355116 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.355083    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:19.721128 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:19.721051    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"]
Oct 31 17:05:19.728338 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:19.728313    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc202bc34_a773_4b3b_b8e8_f11b473518d2.slice/crio-8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0 WatchSource:0}: Error finding container 8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0: Status 404 returned error can't find the container with id 8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0
Oct 31 17:05:20.572391 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:20.572354    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg" event={"ID":"c202bc34-a773-4b3b-b8e8-f11b473518d2","Type":"ContainerStarted","Data":"4538d7a731995fcf60ed67e387d78bde830a5387cada742f2f29caf705427bc6"}
Oct 31 17:05:20.572925 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:20.572418    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg" event={"ID":"c202bc34-a773-4b3b-b8e8-f11b473518d2","Type":"ContainerStarted","Data":"8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0"}
Oct 31 17:05:20.676262 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:20.676212    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg" podStartSLOduration=2.676199139 podStartE2EDuration="2.676199139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:20.675862339 +0000 UTC m=+3765.737437524" watchObservedRunningTime="2026-10-31 17:05:20.676199139 +0000 UTC m=+3765.737774325"
Oct 31 17:05:21.585430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:21.585391    4417 generic.go:356] "Generic (PLEG): container finished" podID="c202bc34-a773-4b3b-b8e8-f11b473518d2" containerID="4538d7a731995fcf60ed67e387d78bde830a5387cada742f2f29caf705427bc6" exitCode=0
Oct 31 17:05:21.585935 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:21.585474    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg" event={"ID":"c202bc34-a773-4b3b-b8e8-f11b473518d2","Type":"ContainerDied","Data":"4538d7a731995fcf60ed67e387d78bde830a5387cada742f2f29caf705427bc6"}
Oct 31 17:05:22.728736 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.728713    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:22.803783 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.803757    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\") pod \"c202bc34-a773-4b3b-b8e8-f11b473518d2\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") "
Oct 31 17:05:22.803908 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.803830    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\" (UniqueName: \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\") pod \"c202bc34-a773-4b3b-b8e8-f11b473518d2\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") "
Oct 31 17:05:22.803954 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.803912    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\") pod \"c202bc34-a773-4b3b-b8e8-f11b473518d2\" (UID: \"c202bc34-a773-4b3b-b8e8-f11b473518d2\") "
Oct 31 17:05:22.804265 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.804236    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca" pod "c202bc34-a773-4b3b-b8e8-f11b473518d2" (UID: "c202bc34-a773-4b3b-b8e8-f11b473518d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:05:22.805830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.805811    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca" pod "c202bc34-a773-4b3b-b8e8-f11b473518d2" (UID: "c202bc34-a773-4b3b-b8e8-f11b473518d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:05:22.805899 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.805882    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv" pod "c202bc34-a773-4b3b-b8e8-f11b473518d2" (UID: "c202bc34-a773-4b3b-b8e8-f11b473518d2"). InnerVolumeSpecName "kube-api-access-6hznv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:05:22.905276 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.905202    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-openshift-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:22.905276 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.905230    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c202bc34-a773-4b3b-b8e8-f11b473518d2-maas-api-xks-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:22.905276 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:22.905240    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hznv\" (UniqueName: \"kubernetes.io/projected/c202bc34-a773-4b3b-b8e8-f11b473518d2-kube-api-access-6hznv\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:23.595740 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:23.595710    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"
Oct 31 17:05:23.595919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:23.595738    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg" event={"ID":"c202bc34-a773-4b3b-b8e8-f11b473518d2","Type":"ContainerDied","Data":"8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0"}
Oct 31 17:05:23.595919 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:23.595784    4417 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e12ab413b7cc808b3b1f79a7811e9cc6ed83ce34239e1f2c3654af3f2a062f0"
Oct 31 17:05:32.270420 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.270384    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:32.270845 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.270795    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c202bc34-a773-4b3b-b8e8-f11b473518d2" containerName="revoke-keys"
Oct 31 17:05:32.270845 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.270807    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c202bc34-a773-4b3b-b8e8-f11b473518d2" containerName="revoke-keys"
Oct 31 17:05:32.270925 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.270885    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c202bc34-a773-4b3b-b8e8-f11b473518d2" containerName="revoke-keys"
Oct 31 17:05:32.274066 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.274047    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.285317 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.285289    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"]
Oct 31 17:05:32.289173 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.289147    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.329446 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.329409    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 17:05:32.329446 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.329431    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 17:05:32.368283 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.368252    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 17:05:32.368469 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.368403    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\""
Oct 31 17:05:32.368469 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.368445    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 17:05:32.371763 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.371740    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:32.374891 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.374872    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"]
Oct 31 17:05:32.392663 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392635    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.392836 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392674    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvsg\" (UniqueName: \"kubernetes.io/projected/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-kube-api-access-jjvsg\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.392836 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392794    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-maas-api-tls\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.392836 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392817    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-metrics-certs\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.392960 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392838    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.392960 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392859    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.393041 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392957    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.393041 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.392977    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9gs\" (UniqueName: \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.493830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.493792    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-maas-api-tls\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.493830 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.493836    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-metrics-certs\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.494067 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.493960    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.494067 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494006    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.494183 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494117    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.494183 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494147    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9gs\" (UniqueName: \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.494301 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494188    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.494301 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494225    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvsg\" (UniqueName: \"kubernetes.io/projected/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-kube-api-access-jjvsg\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.494766 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494738    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.494906 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.494869    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.496504 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.496480    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-metrics-certs\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.496717 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.496700    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.496834 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.496817    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.496902 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.496824    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-maas-api-tls\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.546678 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.546653    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvsg\" (UniqueName: \"kubernetes.io/projected/679f9b33-6b0d-419d-98e5-5ca6f4c0513f-kube-api-access-jjvsg\") pod \"maas-api-67bf6fc4b5-djngp\" (UID: \"679f9b33-6b0d-419d-98e5-5ca6f4c0513f\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.546829 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.546805    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9gs\" (UniqueName: \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\") pod \"maas-api-7bfdbbc5d5-vn89l\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.589518 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.589496    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:32.601810 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.601782    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:32.959608 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:05:32.959514    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:05:32.988229 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.988195    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:32.994894 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:32.994868    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9d7b90_2ed7_4b6b_b28b_b0d541d09978.slice/crio-8689d7d9a336713e09bafbe6b535f05dced96155c78c6a6abe0417202a65ac7a WatchSource:0}: Error finding container 8689d7d9a336713e09bafbe6b535f05dced96155c78c6a6abe0417202a65ac7a: Status 404 returned error can't find the container with id 8689d7d9a336713e09bafbe6b535f05dced96155c78c6a6abe0417202a65ac7a
Oct 31 17:05:32.996340 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:32.996320    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 17:05:33.015210 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:33.015187    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"]
Oct 31 17:05:33.037983 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:33.037956    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679f9b33_6b0d_419d_98e5_5ca6f4c0513f.slice/crio-80a8e724f8644229c62ea2a2621a6af85cdc08432b3e166d6d6a7bf01f376a75 WatchSource:0}: Error finding container 80a8e724f8644229c62ea2a2621a6af85cdc08432b3e166d6d6a7bf01f376a75: Status 404 returned error can't find the container with id 80a8e724f8644229c62ea2a2621a6af85cdc08432b3e166d6d6a7bf01f376a75
Oct 31 17:05:33.056217 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:33.056191    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"]
Oct 31 17:05:33.056333 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:33.056226    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gjn2v"]
Oct 31 17:05:33.643221 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:33.643186    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp" event={"ID":"679f9b33-6b0d-419d-98e5-5ca6f4c0513f","Type":"ContainerStarted","Data":"80a8e724f8644229c62ea2a2621a6af85cdc08432b3e166d6d6a7bf01f376a75"}
Oct 31 17:05:33.644252 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:33.644227    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" event={"ID":"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978","Type":"ContainerStarted","Data":"8689d7d9a336713e09bafbe6b535f05dced96155c78c6a6abe0417202a65ac7a"}
Oct 31 17:05:34.650687 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.650639    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp" event={"ID":"679f9b33-6b0d-419d-98e5-5ca6f4c0513f","Type":"ContainerStarted","Data":"ffe5ed94c757d2fd90a51abe21124e904172e8866f59400f122a9a10433092ef"}
Oct 31 17:05:34.651136 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.650704    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:34.652092 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.652068    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" event={"ID":"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978","Type":"ContainerStarted","Data":"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"}
Oct 31 17:05:34.652164 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.652126    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:34.758244 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.758188    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp" podStartSLOduration=1.8619725919999999 podStartE2EDuration="2.758174914s" totalImagesPullingTime="896.202322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:34.756946084 +0000 UTC m=+3779.818521269" watchObservedRunningTime="2026-10-31 17:05:34.758174914 +0000 UTC m=+3779.819750150"
Oct 31 17:05:34.863982 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:34.863930    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" podStartSLOduration=1.954753702 podStartE2EDuration="2.863914779s" totalImagesPullingTime="909.161077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:34.862085061 +0000 UTC m=+3779.923660259" watchObservedRunningTime="2026-10-31 17:05:34.863914779 +0000 UTC m=+3779.925489965"
Oct 31 17:05:35.048892 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:35.048857    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df" path="/var/lib/kubelet/pods/7c7c8b83-c7ea-4be0-bab7-bd7fce22c8df/volumes"
Oct 31 17:05:39.204678 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.204646    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:05:39.213278 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.213252    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.261287 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.261234    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c6ac3112-openshift-default-dockercfg-575q2\""
Oct 31 17:05:39.312750 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.312703    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:05:39.362679 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362640    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.362928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362776    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.362928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362821    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.362928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362847    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.362928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362874    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.362928 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362921    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjbx\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.363297 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.362986    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.363297 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.363010    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.363297 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.363040    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463641 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463547    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463641 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463633    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463660    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463684    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463709    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463749    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjbx\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463807    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463830    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.463871 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.463857    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.464258 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.464086    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.464258 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.464091    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.464258 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.464241    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.464428 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.464283    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.464723 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.464699    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.466436 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.466414    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.466664 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.466647    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.516375 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.516338    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.516800 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.516775    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjbx\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\") pod \"e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") " pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.526550 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.526506    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:39.949424 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.949384    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:05:39.957463 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:39.957438    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168981f8_2263_43e3_8438_59560ff89300.slice/crio-344979253a122ec0eff97c62bbad2db575bf852cb0e436fa0bc83367c28b4154 WatchSource:0}: Error finding container 344979253a122ec0eff97c62bbad2db575bf852cb0e436fa0bc83367c28b4154: Status 404 returned error can't find the container with id 344979253a122ec0eff97c62bbad2db575bf852cb0e436fa0bc83367c28b4154
Oct 31 17:05:39.959724 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.959691    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:05:39.959812 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.959758    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:05:39.959812 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:39.959789    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:05:40.663490 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.663462    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-djngp"
Oct 31 17:05:40.663969 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.663951    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:40.683112 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.683081    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerStarted","Data":"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"}
Oct 31 17:05:40.683246 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.683127    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerStarted","Data":"344979253a122ec0eff97c62bbad2db575bf852cb0e436fa0bc83367c28b4154"}
Oct 31 17:05:40.815698 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.815668    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:40.815978 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.815943    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" podUID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" containerName="maas-api" containerID="cri-o://46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358" gracePeriod=30
Oct 31 17:05:40.878851 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:40.878797    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podStartSLOduration=1.8787842750000001 podStartE2EDuration="1.878784275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:40.876744909 +0000 UTC m=+3785.938320094" watchObservedRunningTime="2026-10-31 17:05:40.878784275 +0000 UTC m=+3785.940359461"
Oct 31 17:05:41.068430 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.068407    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:41.181301 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.181273    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\" (UniqueName: \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\") pod \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") "
Oct 31 17:05:41.181478 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.181350    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\") pod \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") "
Oct 31 17:05:41.181478 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.181375    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\") pod \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") "
Oct 31 17:05:41.181478 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.181429    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\") pod \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\" (UID: \"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978\") "
Oct 31 17:05:41.181886 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.181800    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca" pod "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" (UID: "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:05:41.183604 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.183544    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs" pod "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" (UID: "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:05:41.183604 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.183571    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs" pod "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" (UID: "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978"). InnerVolumeSpecName "kube-api-access-zm9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:05:41.183758 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.183709    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls" pod "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" (UID: "1d9d7b90-2ed7-4b6b-b28b-b0d541d09978"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:05:41.283162 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.283131    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:41.283162 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.283155    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:41.283162 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.283166    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:41.283395 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.283176    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9gs\" (UniqueName: \"kubernetes.io/projected/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978-kube-api-access-zm9gs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:05:41.527198 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.527111    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:41.532370 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.532342    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:41.689596 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.689565    4417 generic.go:356] "Generic (PLEG): container finished" podID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" containerID="46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358" exitCode=0
Oct 31 17:05:41.690093 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.689607    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"
Oct 31 17:05:41.690093 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.689653    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" event={"ID":"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978","Type":"ContainerDied","Data":"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"}
Oct 31 17:05:41.690093 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.689710    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l" event={"ID":"1d9d7b90-2ed7-4b6b-b28b-b0d541d09978","Type":"ContainerDied","Data":"8689d7d9a336713e09bafbe6b535f05dced96155c78c6a6abe0417202a65ac7a"}
Oct 31 17:05:41.690093 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.689739    4417 scope.go:122] "RemoveContainer" containerID="46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"
Oct 31 17:05:41.690093 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.690042    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:41.691067 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.691037    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:05:41.699857 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.699837    4417 scope.go:122] "RemoveContainer" containerID="46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"
Oct 31 17:05:41.700114 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:05:41.700096    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358\": container with ID starting with 46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358 not found: ID does not exist" containerID="46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"
Oct 31 17:05:41.700212 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.700121    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358"} err="failed to get container status \"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358\": rpc error: code = NotFound desc = could not find container \"46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358\": container with ID starting with 46c95da912e83ddcd54521b17c904ae6c0e97e397770ad509d67357c46c32358 not found: ID does not exist"
Oct 31 17:05:41.839631 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.839589    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:41.841462 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:41.841438    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-vn89l"]
Oct 31 17:05:43.050256 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:43.050211    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" path="/var/lib/kubelet/pods/1d9d7b90-2ed7-4b6b-b28b-b0d541d09978/volumes"
Oct 31 17:05:46.635290 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.635251    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:05:46.635884 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.635861    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" containerName="maas-api"
Oct 31 17:05:46.635977 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.635887    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" containerName="maas-api"
Oct 31 17:05:46.636036 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.635982    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9d7b90-2ed7-4b6b-b28b-b0d541d09978" containerName="maas-api"
Oct 31 17:05:46.640297 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.640275    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.688544 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.688520    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c6ac3112\""
Oct 31 17:05:46.735949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.735889    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbd66\" (UniqueName: \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.736386 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.736357    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.737879 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.737860    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c6ac3112-dockercfg-fw7v7\""
Oct 31 17:05:46.742986 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.742588    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:05:46.799042 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.799017    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:05:46.802586 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.802565    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.837355 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.837333    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.837487 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.837371    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbd66\" (UniqueName: \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.837565 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.837552    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.837648 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.837611    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6msg\" (UniqueName: \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.838234 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.838211    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.859379 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.859360    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:05:46.890050 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.889982    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbd66\" (UniqueName: \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\") pod \"payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.938216 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.938185    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6msg\" (UniqueName: \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.938311 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.938225    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.938851 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.938834    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:46.953077 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.953055    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:46.992035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:46.990850    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6msg\" (UniqueName: \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\") pod \"payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") " pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:47.114955 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.114922    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:47.319672 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.319599    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:05:47.326362 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:47.326337    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc564b951_c986_4b6a_8e68_1166fdd8a737.slice/crio-fe1087f7c542ff74b83c9bdf79be12ce824e13198da922e277fccd64e2f58bac WatchSource:0}: Error finding container fe1087f7c542ff74b83c9bdf79be12ce824e13198da922e277fccd64e2f58bac: Status 404 returned error can't find the container with id fe1087f7c542ff74b83c9bdf79be12ce824e13198da922e277fccd64e2f58bac
Oct 31 17:05:47.479470 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.479412    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:05:47.486134 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:47.486107    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a25e78_60b0_49e5_8550_50a4d8a3a3b9.slice/crio-6d8ad21de0c3e071d13a13f13a585e23b98907d03955cc98be1c4bb3ce419f69 WatchSource:0}: Error finding container 6d8ad21de0c3e071d13a13f13a585e23b98907d03955cc98be1c4bb3ce419f69: Status 404 returned error can't find the container with id 6d8ad21de0c3e071d13a13f13a585e23b98907d03955cc98be1c4bb3ce419f69
Oct 31 17:05:47.722253 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.722211    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" event={"ID":"80a25e78-60b0-49e5-8550-50a4d8a3a3b9","Type":"ContainerStarted","Data":"c26634b19417da6e11aa7437441d72a3d919a8ce1f78bddd0007d7138b9bd18a"}
Oct 31 17:05:47.722717 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.722274    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" event={"ID":"80a25e78-60b0-49e5-8550-50a4d8a3a3b9","Type":"ContainerStarted","Data":"6d8ad21de0c3e071d13a13f13a585e23b98907d03955cc98be1c4bb3ce419f69"}
Oct 31 17:05:47.722717 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.722307    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:47.723807 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.723785    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" event={"ID":"c564b951-c986-4b6a-8e68-1166fdd8a737","Type":"ContainerStarted","Data":"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"}
Oct 31 17:05:47.723913 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.723822    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" event={"ID":"c564b951-c986-4b6a-8e68-1166fdd8a737","Type":"ContainerStarted","Data":"fe1087f7c542ff74b83c9bdf79be12ce824e13198da922e277fccd64e2f58bac"}
Oct 31 17:05:47.723913 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.723882    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:05:47.827482 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.827440    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" podStartSLOduration=1.827426641 podStartE2EDuration="1.827426641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:47.825827535 +0000 UTC m=+3792.887402721" watchObservedRunningTime="2026-10-31 17:05:47.827426641 +0000 UTC m=+3792.889001821"
Oct 31 17:05:47.932190 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:47.932137    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" podStartSLOduration=1.93212439 podStartE2EDuration="1.93212439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:47.930490347 +0000 UTC m=+3792.992065533" watchObservedRunningTime="2026-10-31 17:05:47.93212439 +0000 UTC m=+3792.993699622"
Oct 31 17:05:48.824225 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.824192    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:05:48.827829 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.827793    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.856421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.856392    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.856546 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.856435    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.856546 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.856537    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.856649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.856563    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkf8\" (UniqueName: \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.875926 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.875904    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c6ac3112-serving-cert\""
Oct 31 17:05:48.923861 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.923836    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:05:48.957611 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.957575    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.957611 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.957609    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkf8\" (UniqueName: \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.957838 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.957706    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.957838 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.957737    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.957838 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:05:48.957818    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-serving-cert: secret "maas-api-e2e-ait-c6ac3112-serving-cert" not found
Oct 31 17:05:48.958020 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:05:48.957876    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls podName:1b0bb818-19da-443a-92d8-66068b087c64 nodeName:}" failed. No retries permitted until 2026-10-31 17:05:49.457859052 +0000 UTC m=+3794.519434216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls") pod "maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" (UID: "1b0bb818-19da-443a-92d8-66068b087c64") : secret "maas-api-e2e-ait-c6ac3112-serving-cert" not found
Oct 31 17:05:48.958393 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.958370    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:48.959960 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:48.959935    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:49.010923 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:49.010893    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkf8\" (UniqueName: \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:49.468068 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:49.468032    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:49.470400 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:49.470377    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") pod \"maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:49.741590 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:49.741495    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:50.107826 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:50.107797    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:05:50.114365 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:05:50.114334    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0bb818_19da_443a_92d8_66068b087c64.slice/crio-e3d779f091407a8a497871ed789680e6e0cd92c8775fd39c7d5b1bd439cc7f51 WatchSource:0}: Error finding container e3d779f091407a8a497871ed789680e6e0cd92c8775fd39c7d5b1bd439cc7f51: Status 404 returned error can't find the container with id e3d779f091407a8a497871ed789680e6e0cd92c8775fd39c7d5b1bd439cc7f51
Oct 31 17:05:50.740963 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:50.740925    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" event={"ID":"1b0bb818-19da-443a-92d8-66068b087c64","Type":"ContainerStarted","Data":"e3d779f091407a8a497871ed789680e6e0cd92c8775fd39c7d5b1bd439cc7f51"}
Oct 31 17:05:51.747864 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:51.747827    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" event={"ID":"1b0bb818-19da-443a-92d8-66068b087c64","Type":"ContainerStarted","Data":"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"}
Oct 31 17:05:51.748358 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:51.748010    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:51.861112 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:51.861058    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" podStartSLOduration=2.964611263 podStartE2EDuration="3.861041483s" totalImagesPullingTime="896.43022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:51.858224051 +0000 UTC m=+3796.919799237" watchObservedRunningTime="2026-10-31 17:05:51.861041483 +0000 UTC m=+3796.922616671"
Oct 31 17:05:57.764264 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:57.764230    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:05:58.731058 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:58.731030    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:05:58.731237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:05:58.731094    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:06:02.061302 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:02.061259    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"]
Oct 31 17:06:02.086939 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:02.086905    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6wb2c"]
Oct 31 17:06:02.943298 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:02.943264    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:06:03.049001 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:03.048972    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac8910f-ebcf-4680-8164-db66ff974b13" path="/var/lib/kubelet/pods/3ac8910f-ebcf-4680-8164-db66ff974b13/volumes"
Oct 31 17:06:06.825709 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.825674    4417 generic.go:356] "Generic (PLEG): container finished" podID="168981f8-2263-43e3-8438-59560ff89300" containerID="1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f" exitCode=0
Oct 31 17:06:06.826144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.825749    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerDied","Data":"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"}
Oct 31 17:06:06.826144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.826104    4417 scope.go:122] "RemoveContainer" containerID="1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"
Oct 31 17:06:06.827362 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.827335    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:06.827424 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.827404    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:06.827461 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:06.827436    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:07.859066 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:07.859022    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerStarted","Data":"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"}
Oct 31 17:06:08.527693 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:08.527656    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:08.529024 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:08.528993    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 17:06:08.529178 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:08.529065    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 17:06:09.526956 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:09.526923    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 17:06:09.527516 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:09.527003    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 17:06:10.527131 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:10.527097    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 17:06:10.527565 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:10.527149    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 17:06:11.527596 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:11.527562    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 17:06:11.528018 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:11.527649    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 17:06:11.690888 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:11.690846    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:12.527427 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:12.527390    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 17:06:12.527629 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:12.527452    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 17:06:13.530779 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:13.530743    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:13.531878 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:13.531858    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:30.831152 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:30.831117    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:06:30.929222 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:30.929189    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:06:31.359692 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.359656    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:06:31.359934 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.359905    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" podUID="1b0bb818-19da-443a-92d8-66068b087c64" containerName="maas-api" containerID="cri-o://d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6" gracePeriod=30
Oct 31 17:06:31.621461 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.621400    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:06:31.696010 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.695980    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") pod \"1b0bb818-19da-443a-92d8-66068b087c64\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") "
Oct 31 17:06:31.696159 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.696039    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\") pod \"1b0bb818-19da-443a-92d8-66068b087c64\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") "
Oct 31 17:06:31.696231 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.696199    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\" (UniqueName: \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\") pod \"1b0bb818-19da-443a-92d8-66068b087c64\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") "
Oct 31 17:06:31.696305 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.696247    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\") pod \"1b0bb818-19da-443a-92d8-66068b087c64\" (UID: \"1b0bb818-19da-443a-92d8-66068b087c64\") "
Oct 31 17:06:31.696673 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.696647    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca" pod "1b0bb818-19da-443a-92d8-66068b087c64" (UID: "1b0bb818-19da-443a-92d8-66068b087c64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:06:31.698091 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.698069    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls" pod "1b0bb818-19da-443a-92d8-66068b087c64" (UID: "1b0bb818-19da-443a-92d8-66068b087c64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:06:31.698195 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.698112    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8" pod "1b0bb818-19da-443a-92d8-66068b087c64" (UID: "1b0bb818-19da-443a-92d8-66068b087c64"). InnerVolumeSpecName "kube-api-access-dzkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:31.698195 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.698128    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs" pod "1b0bb818-19da-443a-92d8-66068b087c64" (UID: "1b0bb818-19da-443a-92d8-66068b087c64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:06:31.797995 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.797961    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:31.797995 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.797989    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0bb818-19da-443a-92d8-66068b087c64-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:31.798178 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.798003    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkf8\" (UniqueName: \"kubernetes.io/projected/1b0bb818-19da-443a-92d8-66068b087c64-kube-api-access-dzkf8\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:31.798178 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.798018    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0bb818-19da-443a-92d8-66068b087c64-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:31.974651 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.974558    4417 generic.go:356] "Generic (PLEG): container finished" podID="1b0bb818-19da-443a-92d8-66068b087c64" containerID="d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6" exitCode=0
Oct 31 17:06:31.975171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.974649    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"
Oct 31 17:06:31.975171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.974655    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" event={"ID":"1b0bb818-19da-443a-92d8-66068b087c64","Type":"ContainerDied","Data":"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"}
Oct 31 17:06:31.975171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.974738    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4" event={"ID":"1b0bb818-19da-443a-92d8-66068b087c64","Type":"ContainerDied","Data":"e3d779f091407a8a497871ed789680e6e0cd92c8775fd39c7d5b1bd439cc7f51"}
Oct 31 17:06:31.975171 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.974772    4417 scope.go:122] "RemoveContainer" containerID="d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"
Oct 31 17:06:31.985376 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.985357    4417 scope.go:122] "RemoveContainer" containerID="d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"
Oct 31 17:06:31.985681 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:31.985661    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6\": container with ID starting with d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6 not found: ID does not exist" containerID="d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"
Oct 31 17:06:31.985773 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:31.985690    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6"} err="failed to get container status \"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6\": rpc error: code = NotFound desc = could not find container \"d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6\": container with ID starting with d4bbf2c955adbad5ad2e1528f04894ce118b4fc9224860fc6f4f4996699b3bb6 not found: ID does not exist"
Oct 31 17:06:32.124698 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:32.124669    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:06:32.128139 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:32.128112    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c6ac3112-7dfd656d55-zc2h4"]
Oct 31 17:06:32.943745 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:32.943707    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:06:33.050576 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:33.050541    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0bb818-19da-443a-92d8-66068b087c64" path="/var/lib/kubelet/pods/1b0bb818-19da-443a-92d8-66068b087c64/volumes"
Oct 31 17:06:35.832352 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:35.832291    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" podUID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" containerName="payload-processing" containerID="cri-o://c26634b19417da6e11aa7437441d72a3d919a8ce1f78bddd0007d7138b9bd18a" gracePeriod=25
Oct 31 17:06:35.930248 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:35.930200    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" podUID="c564b951-c986-4b6a-8e68-1166fdd8a737" containerName="payload-pre-processing" containerID="cri-o://5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b" gracePeriod=25
Oct 31 17:06:36.015179 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.015145    4417 generic.go:356] "Generic (PLEG): container finished" podID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" containerID="c26634b19417da6e11aa7437441d72a3d919a8ce1f78bddd0007d7138b9bd18a" exitCode=0
Oct 31 17:06:36.015442 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.015206    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" event={"ID":"80a25e78-60b0-49e5-8550-50a4d8a3a3b9","Type":"ContainerDied","Data":"c26634b19417da6e11aa7437441d72a3d919a8ce1f78bddd0007d7138b9bd18a"}
Oct 31 17:06:36.092707 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.092641    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:06:36.139161 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.139129    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\") pod \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") "
Oct 31 17:06:36.139317 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.139191    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\" (UniqueName: \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\") pod \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\" (UID: \"80a25e78-60b0-49e5-8550-50a4d8a3a3b9\") "
Oct 31 17:06:36.139565 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.139541    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume" pod "80a25e78-60b0-49e5-8550-50a4d8a3a3b9" (UID: "80a25e78-60b0-49e5-8550-50a4d8a3a3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:06:36.141727 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.141705    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg" pod "80a25e78-60b0-49e5-8550-50a4d8a3a3b9" (UID: "80a25e78-60b0-49e5-8550-50a4d8a3a3b9"). InnerVolumeSpecName "kube-api-access-x6msg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:36.179078 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.179057    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:06:36.240150 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.240114    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\" (UniqueName: \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\") pod \"c564b951-c986-4b6a-8e68-1166fdd8a737\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") "
Oct 31 17:06:36.240318 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.240224    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\") pod \"c564b951-c986-4b6a-8e68-1166fdd8a737\" (UID: \"c564b951-c986-4b6a-8e68-1166fdd8a737\") "
Oct 31 17:06:36.240563 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.240538    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:36.240563 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.240550    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume" pod "c564b951-c986-4b6a-8e68-1166fdd8a737" (UID: "c564b951-c986-4b6a-8e68-1166fdd8a737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:06:36.240563 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.240565    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6msg\" (UniqueName: \"kubernetes.io/projected/80a25e78-60b0-49e5-8550-50a4d8a3a3b9-kube-api-access-x6msg\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:36.242351 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.242328    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66" pod "c564b951-c986-4b6a-8e68-1166fdd8a737" (UID: "c564b951-c986-4b6a-8e68-1166fdd8a737"). InnerVolumeSpecName "kube-api-access-dbd66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:36.341528 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.341498    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbd66\" (UniqueName: \"kubernetes.io/projected/c564b951-c986-4b6a-8e68-1166fdd8a737-kube-api-access-dbd66\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:36.341528 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.341526    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c564b951-c986-4b6a-8e68-1166fdd8a737-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:36.957200 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:36.957166    4417 scope.go:122] "RemoveContainer" containerID="885b23aaf733e51bac6b09f2c278ed77247d5f7c042761b066014e4b057e700b"
Oct 31 17:06:37.020780 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.020756    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"
Oct 31 17:06:37.020908 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.020798    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf" event={"ID":"80a25e78-60b0-49e5-8550-50a4d8a3a3b9","Type":"ContainerDied","Data":"6d8ad21de0c3e071d13a13f13a585e23b98907d03955cc98be1c4bb3ce419f69"}
Oct 31 17:06:37.020908 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.020865    4417 scope.go:122] "RemoveContainer" containerID="c26634b19417da6e11aa7437441d72a3d919a8ce1f78bddd0007d7138b9bd18a"
Oct 31 17:06:37.022075 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.022053    4417 generic.go:356] "Generic (PLEG): container finished" podID="c564b951-c986-4b6a-8e68-1166fdd8a737" containerID="5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b" exitCode=0
Oct 31 17:06:37.022185 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.022106    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"
Oct 31 17:06:37.022185 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.022106    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" event={"ID":"c564b951-c986-4b6a-8e68-1166fdd8a737","Type":"ContainerDied","Data":"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"}
Oct 31 17:06:37.022275 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.022223    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c" event={"ID":"c564b951-c986-4b6a-8e68-1166fdd8a737","Type":"ContainerDied","Data":"fe1087f7c542ff74b83c9bdf79be12ce824e13198da922e277fccd64e2f58bac"}
Oct 31 17:06:37.030150 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.030126    4417 scope.go:122] "RemoveContainer" containerID="5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"
Oct 31 17:06:37.037660 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.037642    4417 scope.go:122] "RemoveContainer" containerID="5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"
Oct 31 17:06:37.037937 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:37.037921    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b\": container with ID starting with 5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b not found: ID does not exist" containerID="5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"
Oct 31 17:06:37.037996 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.037945    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b"} err="failed to get container status \"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b\": rpc error: code = NotFound desc = could not find container \"5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b\": container with ID starting with 5c2fe9794c4debb2bd375f3859632c8c14c9be3d13660748fc85021e17ee412b not found: ID does not exist"
Oct 31 17:06:37.171291 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.171211    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:06:37.216905 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.216876    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c6ac3112-55976f95c5-hk97c"]
Oct 31 17:06:37.275687 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.275656    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:06:37.279385 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:37.279359    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c6ac3112-f6678b99f-wdvpf"]
Oct 31 17:06:39.047519 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:39.047481    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" path="/var/lib/kubelet/pods/80a25e78-60b0-49e5-8550-50a4d8a3a3b9/volumes"
Oct 31 17:06:39.048009 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:39.047993    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c564b951-c986-4b6a-8e68-1166fdd8a737" path="/var/lib/kubelet/pods/c564b951-c986-4b6a-8e68-1166fdd8a737/volumes"
Oct 31 17:06:40.371184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:40.371151    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:06:40.371571 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:40.371342    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" containerID="cri-o://1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5" gracePeriod=30
Oct 31 17:06:42.043709 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.043668    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:06:42.044187 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044167    4417 cpu_manager.go:414] "containerMap: removing container" podUID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" containerName="payload-processing"
Oct 31 17:06:42.044251 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044191    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" containerName="payload-processing"
Oct 31 17:06:42.044251 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044204    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1b0bb818-19da-443a-92d8-66068b087c64" containerName="maas-api"
Oct 31 17:06:42.044251 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044211    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0bb818-19da-443a-92d8-66068b087c64" containerName="maas-api"
Oct 31 17:06:42.044354 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044265    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c564b951-c986-4b6a-8e68-1166fdd8a737" containerName="payload-pre-processing"
Oct 31 17:06:42.044354 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044275    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c564b951-c986-4b6a-8e68-1166fdd8a737" containerName="payload-pre-processing"
Oct 31 17:06:42.044421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044361    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c564b951-c986-4b6a-8e68-1166fdd8a737" containerName="payload-pre-processing"
Oct 31 17:06:42.044421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044376    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a25e78-60b0-49e5-8550-50a4d8a3a3b9" containerName="payload-processing"
Oct 31 17:06:42.044421 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.044390    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0bb818-19da-443a-92d8-66068b087c64" containerName="maas-api"
Oct 31 17:06:42.047968 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.047949    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095278 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095253    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095406 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095318    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095406 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095367    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095513 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095424    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095513 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095446    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095567    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxq4\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095590    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095778 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095670    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.095778 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.095693    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.138132 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.138107    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-713882-openshift-default-dockercfg-94ls2\""
Oct 31 17:06:42.150558 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.150538    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:06:42.197049 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197018    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197049 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197051    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197081    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197102    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197133    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxq4\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197157    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197191    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197213    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197280 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197281    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197727 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197702    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.197935 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197725    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.198021 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197791    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.198021 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197942    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.198021 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.197942    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.199420 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.199397    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.199703 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.199686    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.250312 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.250285    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.250794 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.250776    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxq4\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\") pod \"e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") " pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.361208 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.361112    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:42.780041 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.779963    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:06:42.786385 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:06:42.786357    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1712191b_d721_4426_855e_40a22e6c566a.slice/crio-0979394b1c755b7dbb603b391043e186e2cc3d57a5ed9e43bb096df95f44fa4d WatchSource:0}: Error finding container 0979394b1c755b7dbb603b391043e186e2cc3d57a5ed9e43bb096df95f44fa4d: Status 404 returned error can't find the container with id 0979394b1c755b7dbb603b391043e186e2cc3d57a5ed9e43bb096df95f44fa4d
Oct 31 17:06:42.788558 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.788529    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:42.788649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.788595    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:42.788649 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:42.788640    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:06:43.057344 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.057301    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerStarted","Data":"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"}
Oct 31 17:06:43.057792 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.057354    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerStarted","Data":"0979394b1c755b7dbb603b391043e186e2cc3d57a5ed9e43bb096df95f44fa4d"}
Oct 31 17:06:43.164880 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.164831    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podStartSLOduration=1.164817751 podStartE2EDuration="1.164817751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:06:43.164129113 +0000 UTC m=+3848.225704299" watchObservedRunningTime="2026-10-31 17:06:43.164817751 +0000 UTC m=+3848.226392935"
Oct 31 17:06:43.361969 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.361873    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:43.366891 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.366867    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:43.532947 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.532916    4417 patch_prober.go:29] interesting pod/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 17:06:43.533100 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:43.532963    4417 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 17:06:44.062873 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:44.062846    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:44.063685 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:44.063665    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:06:45.622712 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.622683    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:45.732979 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.732892    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.732979 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.732938    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.732979 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.732978    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733024    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733131    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733163    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733203    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:06:45.733269 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733248    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733279    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733292    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:06:45.733520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733309    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\") pod \"168981f8-2263-43e3-8438-59560ff89300\" (UID: \"168981f8-2263-43e3-8438-59560ff89300\") "
Oct 31 17:06:45.733520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733429    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:06:45.733901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733660    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:06:45.733901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733675    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.733901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733701    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.733901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733713    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.733901 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.733805    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:06:45.735345 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.735320    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 17:06:45.735578 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.735562    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:06:45.735757 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.735742    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "kube-api-access-swjbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:45.736170 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.736143    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token" pod "168981f8-2263-43e3-8438-59560ff89300" (UID: "168981f8-2263-43e3-8438-59560ff89300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:06:45.834237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834208    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.834237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834233    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168981f8-2263-43e3-8438-59560ff89300-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.834237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834242    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168981f8-2263-43e3-8438-59560ff89300-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.834237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834252    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168981f8-2263-43e3-8438-59560ff89300-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.834520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834261    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:45.834520 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:45.834271    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjbx\" (UniqueName: \"kubernetes.io/projected/168981f8-2263-43e3-8438-59560ff89300-kube-api-access-swjbx\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:06:46.072945 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.072912    4417 generic.go:356] "Generic (PLEG): container finished" podID="168981f8-2263-43e3-8438-59560ff89300" containerID="1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5" exitCode=0
Oct 31 17:06:46.073118 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.072962    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"
Oct 31 17:06:46.073118 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.073002    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerDied","Data":"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"}
Oct 31 17:06:46.073118 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.073045    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q" event={"ID":"168981f8-2263-43e3-8438-59560ff89300","Type":"ContainerDied","Data":"344979253a122ec0eff97c62bbad2db575bf852cb0e436fa0bc83367c28b4154"}
Oct 31 17:06:46.073118 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.073066    4417 scope.go:122] "RemoveContainer" containerID="1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"
Oct 31 17:06:46.083264 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.083248    4417 scope.go:122] "RemoveContainer" containerID="1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"
Oct 31 17:06:46.091331 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.091318    4417 scope.go:122] "RemoveContainer" containerID="1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"
Oct 31 17:06:46.091563 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:46.091543    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5\": container with ID starting with 1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5 not found: ID does not exist" containerID="1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"
Oct 31 17:06:46.091632 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.091571    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5"} err="failed to get container status \"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5\": rpc error: code = NotFound desc = could not find container \"1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5\": container with ID starting with 1352aa17e226bd91909adf8820727f29eb8bb73aa86dc41afc1bc389700051b5 not found: ID does not exist"
Oct 31 17:06:46.091632 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.091587    4417 scope.go:122] "RemoveContainer" containerID="1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"
Oct 31 17:06:46.091818 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:46.091803    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f\": container with ID starting with 1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f not found: ID does not exist" containerID="1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"
Oct 31 17:06:46.091864 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.091822    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f"} err="failed to get container status \"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f\": rpc error: code = NotFound desc = could not find container \"1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f\": container with ID starting with 1120b491c88f9fc979158878b22d53a5f7060080d3e2304829b7bcbc4ab64e1f not found: ID does not exist"
Oct 31 17:06:46.230679 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.230646    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:06:46.233199 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:46.233164    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c6ac3112-openshift-default-6875446d77-kc77q"]
Oct 31 17:06:47.048228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:47.048186    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168981f8-2263-43e3-8438-59560ff89300" path="/var/lib/kubelet/pods/168981f8-2263-43e3-8438-59560ff89300/volumes"
Oct 31 17:06:49.541643 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.541588    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:06:49.542043 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.542027    4417 cpu_manager.go:414] "containerMap: removing container" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.542043 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.542040    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.542144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.542057    4417 cpu_manager.go:414] "containerMap: removing container" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.542144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.542064    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.542144 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.542128    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.546818 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.546798    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.595300 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.595280    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-713882\""
Oct 31 17:06:49.641379 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.641349    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-713882-dockercfg-fpf7t\""
Oct 31 17:06:49.645345 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.645302    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:06:49.671412 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.671380    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5djjw\" (UniqueName: \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.671571 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.671513    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.710033 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.710005    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:06:49.710446 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.710435    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="168981f8-2263-43e3-8438-59560ff89300" containerName="istio-proxy"
Oct 31 17:06:49.713477 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.713451    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.771022 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.770992    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:06:49.772566 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.772544    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.772628 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.772602    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.772676 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.772665    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsvj\" (UniqueName: \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.772737 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.772721    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5djjw\" (UniqueName: \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.773262 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.773243    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.824998 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.824926    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5djjw\" (UniqueName: \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\") pod \"payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.859802 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.859775    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:49.874343 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.874299    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.874455 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.874391    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsvj\" (UniqueName: \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.875026 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.875003    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:49.927611 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:49.927183    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsvj\" (UniqueName: \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\") pod \"payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:50.026448 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:50.026413    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:50.226231 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:50.226197    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:06:50.233147 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:06:50.233117    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc938ccbf_0ba5_4e1c_bce3_65a3ac7087fa.slice/crio-82d44711bd1909653c6161b4c30d462a7007bdfc3354f53ad10a46193e3ba17b WatchSource:0}: Error finding container 82d44711bd1909653c6161b4c30d462a7007bdfc3354f53ad10a46193e3ba17b: Status 404 returned error can't find the container with id 82d44711bd1909653c6161b4c30d462a7007bdfc3354f53ad10a46193e3ba17b
Oct 31 17:06:50.392737 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:50.392669    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:06:50.399435 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:06:50.399406    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c0b3f_387a_4f53_a0f3_87d04c736e4b.slice/crio-72a3ef0b49d1f650693c1214bd0052c4f7edf513a6c48a62622f1d1ba48d1b61 WatchSource:0}: Error finding container 72a3ef0b49d1f650693c1214bd0052c4f7edf513a6c48a62622f1d1ba48d1b61: Status 404 returned error can't find the container with id 72a3ef0b49d1f650693c1214bd0052c4f7edf513a6c48a62622f1d1ba48d1b61
Oct 31 17:06:51.099213 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.099176    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" event={"ID":"070c0b3f-387a-4f53-a0f3-87d04c736e4b","Type":"ContainerStarted","Data":"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"}
Oct 31 17:06:51.099667 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.099231    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" event={"ID":"070c0b3f-387a-4f53-a0f3-87d04c736e4b","Type":"ContainerStarted","Data":"72a3ef0b49d1f650693c1214bd0052c4f7edf513a6c48a62622f1d1ba48d1b61"}
Oct 31 17:06:51.099667 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.099289    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:06:51.100645 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.100605    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" event={"ID":"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa","Type":"ContainerStarted","Data":"cd37fd9d5f53ed581f6972686136e56d9044e468d15f3258bf2742945ac66efb"}
Oct 31 17:06:51.100751 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.100662    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" event={"ID":"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa","Type":"ContainerStarted","Data":"82d44711bd1909653c6161b4c30d462a7007bdfc3354f53ad10a46193e3ba17b"}
Oct 31 17:06:51.100790 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.100761    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:06:51.204083 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.204037    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" podStartSLOduration=2.204026679 podStartE2EDuration="2.204026679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:06:51.203095263 +0000 UTC m=+3856.264670449" watchObservedRunningTime="2026-10-31 17:06:51.204026679 +0000 UTC m=+3856.265601863"
Oct 31 17:06:51.307806 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:51.307762    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" podStartSLOduration=2.307749598 podStartE2EDuration="2.307749598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:06:51.307001927 +0000 UTC m=+3856.368577111" watchObservedRunningTime="2026-10-31 17:06:51.307749598 +0000 UTC m=+3856.369324783"
Oct 31 17:06:52.710129 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.710096    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:06:52.713718 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.713687    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.762114 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.762091    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-713882-serving-cert\""
Oct 31 17:06:52.801218 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.801189    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdxb\" (UniqueName: \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.801350 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.801253    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.801396 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.801378    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.801433 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.801411    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.810261 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.810234    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:06:52.902750 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.902714    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.902949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.902758    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.902949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.902805    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdxb\" (UniqueName: \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.902949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.902845    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.903123 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:52.902966    4417 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-serving-cert: secret "maas-api-e2e-authcfg-http-713882-serving-cert" not found
Oct 31 17:06:52.903123 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:06:52.903048    4417 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls podName:b6a3e7f0-2d2d-4840-85e4-6600011149dc nodeName:}" failed. No retries permitted until 2026-10-31 17:06:53.403027019 +0000 UTC m=+3858.464602202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls") pod "maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" (UID: "b6a3e7f0-2d2d-4840-85e4-6600011149dc") : secret "maas-api-e2e-authcfg-http-713882-serving-cert" not found
Oct 31 17:06:52.903544 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.903520    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.905356 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.905334    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:52.955543 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:52.955511    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdxb\" (UniqueName: \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:53.409026 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:53.408995    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:53.411375 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:53.411347    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:53.626457 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:53.626422    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:53.993347 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:53.993256    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:06:54.000898 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:06:54.000872    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a3e7f0_2d2d_4840_85e4_6600011149dc.slice/crio-833c9f372d8e7c7a3f53b5046a3a7e6adc5c5d3e79fd418851c64abf09c926b0 WatchSource:0}: Error finding container 833c9f372d8e7c7a3f53b5046a3a7e6adc5c5d3e79fd418851c64abf09c926b0: Status 404 returned error can't find the container with id 833c9f372d8e7c7a3f53b5046a3a7e6adc5c5d3e79fd418851c64abf09c926b0
Oct 31 17:06:54.115723 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:54.115688    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" event={"ID":"b6a3e7f0-2d2d-4840-85e4-6600011149dc","Type":"ContainerStarted","Data":"833c9f372d8e7c7a3f53b5046a3a7e6adc5c5d3e79fd418851c64abf09c926b0"}
Oct 31 17:06:55.122244 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:55.122205    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" event={"ID":"b6a3e7f0-2d2d-4840-85e4-6600011149dc","Type":"ContainerStarted","Data":"6d971b349f3c98614e62aa80ca8ad4cae4f1b406d19c4a3fdc179201a199c18b"}
Oct 31 17:06:55.122672 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:55.122272    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:06:55.227825 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:06:55.227739    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" podStartSLOduration=2.311367233 podStartE2EDuration="3.227726359s" totalImagesPullingTime="916.359126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:06:55.227158858 +0000 UTC m=+3860.288734043" watchObservedRunningTime="2026-10-31 17:06:55.227726359 +0000 UTC m=+3860.289301542"
Oct 31 17:07:01.132543 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:01.132514    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:07:02.108564 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:02.108533    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:07:02.108754 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:02.108604    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:07:02.968409 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:07:02.968373    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:07:09.204529 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.204496    4417 generic.go:356] "Generic (PLEG): container finished" podID="1712191b-d721-4426-855e-40a22e6c566a" containerID="1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7" exitCode=0
Oct 31 17:07:09.204992 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.204569    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerDied","Data":"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"}
Oct 31 17:07:09.204992 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.204933    4417 scope.go:122] "RemoveContainer" containerID="1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"
Oct 31 17:07:09.206193 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.206163    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:07:09.206262 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.206235    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:07:09.206307 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:09.206265    4417 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 17:07:10.212124 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:10.212094    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerStarted","Data":"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"}
Oct 31 17:07:10.362039 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:10.361996    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:07:10.363253 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:10.363224    4417 patch_prober.go:29] interesting pod/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 17:07:10.363389 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:10.363281    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 17:07:11.362236 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:11.362196    4417 patch_prober.go:29] interesting pod/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 17:07:11.362657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:11.362265    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 17:07:12.361708 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:12.361664    4417 patch_prober.go:29] interesting pod/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 17:07:12.361949 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:12.361747    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 17:07:13.362197 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:13.362150    4417 patch_prober.go:29] interesting pod/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 17:07:13.362738 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:13.362214    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 17:07:14.063770 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:14.063735    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:07:14.362229 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:14.362140    4417 patch_prober.go:29] interesting pod/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 17:07:14.362229 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:14.362195    4417 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 17:07:15.366502 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:15.366472    4417 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:07:15.367595 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:15.367544    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:07:32.942917 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:07:32.942871    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:07:36.984331 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:07:36.984297    4417 scope.go:122] "RemoveContainer" containerID="be0eda95ed89198fdb84e4f95fc34648678f8106f77ce08735c57bcd0fc1bfd2"
Oct 31 17:08:02.943335 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:08:02.943249    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:08:32.942590 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:08:32.942550    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:08:42.349489 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:42.349449    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:08:42.450595 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:42.450560    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:08:47.350395 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.350340    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" podUID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" containerName="payload-processing" containerID="cri-o://64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94" gracePeriod=25
Oct 31 17:08:47.451756 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.451674    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" podUID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" containerName="payload-pre-processing" containerID="cri-o://cd37fd9d5f53ed581f6972686136e56d9044e468d15f3258bf2742945ac66efb" gracePeriod=25
Oct 31 17:08:47.600000 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.599973    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:08:47.665761 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.665721    4417 generic.go:356] "Generic (PLEG): container finished" podID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" containerID="cd37fd9d5f53ed581f6972686136e56d9044e468d15f3258bf2742945ac66efb" exitCode=0
Oct 31 17:08:47.665941 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.665793    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" event={"ID":"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa","Type":"ContainerDied","Data":"cd37fd9d5f53ed581f6972686136e56d9044e468d15f3258bf2742945ac66efb"}
Oct 31 17:08:47.667287 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.667264    4417 generic.go:356] "Generic (PLEG): container finished" podID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" containerID="64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94" exitCode=0
Oct 31 17:08:47.667425 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.667306    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" event={"ID":"070c0b3f-387a-4f53-a0f3-87d04c736e4b","Type":"ContainerDied","Data":"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"}
Oct 31 17:08:47.667425 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.667345    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2" event={"ID":"070c0b3f-387a-4f53-a0f3-87d04c736e4b","Type":"ContainerDied","Data":"72a3ef0b49d1f650693c1214bd0052c4f7edf513a6c48a62622f1d1ba48d1b61"}
Oct 31 17:08:47.667425 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.667309    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"
Oct 31 17:08:47.667598 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.667373    4417 scope.go:122] "RemoveContainer" containerID="64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"
Oct 31 17:08:47.686997 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.686978    4417 scope.go:122] "RemoveContainer" containerID="64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"
Oct 31 17:08:47.687348 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:08:47.687328    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94\": container with ID starting with 64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94 not found: ID does not exist" containerID="64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"
Oct 31 17:08:47.687455 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.687360    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94"} err="failed to get container status \"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94\": rpc error: code = NotFound desc = could not find container \"64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94\": container with ID starting with 64408defe128db99a25814f21c6cec9f8d77135e9741924c3f0c0c0b1f5b1c94 not found: ID does not exist"
Oct 31 17:08:47.691208 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.691190    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:08:47.697292 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.697272    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\" (UniqueName: \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\") pod \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") "
Oct 31 17:08:47.697387 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.697319    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\") pod \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\" (UID: \"070c0b3f-387a-4f53-a0f3-87d04c736e4b\") "
Oct 31 17:08:47.697679 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.697660    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume" pod "070c0b3f-387a-4f53-a0f3-87d04c736e4b" (UID: "070c0b3f-387a-4f53-a0f3-87d04c736e4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:08:47.699214 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.699197    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj" pod "070c0b3f-387a-4f53-a0f3-87d04c736e4b" (UID: "070c0b3f-387a-4f53-a0f3-87d04c736e4b"). InnerVolumeSpecName "kube-api-access-6hsvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:08:47.798774 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.798744    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\") pod \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") "
Oct 31 17:08:47.798952 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.798801    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\" (UniqueName: \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\") pod \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\" (UID: \"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa\") "
Oct 31 17:08:47.799055 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.799032    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hsvj\" (UniqueName: \"kubernetes.io/projected/070c0b3f-387a-4f53-a0f3-87d04c736e4b-kube-api-access-6hsvj\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:47.799055 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.799050    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/070c0b3f-387a-4f53-a0f3-87d04c736e4b-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:47.799196 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.799172    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume" pod "c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" (UID: "c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:08:47.800769 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.800750    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw" pod "c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" (UID: "c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa"). InnerVolumeSpecName "kube-api-access-5djjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:08:47.899578 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.899499    4417 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-plugins-config-volume\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:47.899578 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:47.899524    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5djjw\" (UniqueName: \"kubernetes.io/projected/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa-kube-api-access-5djjw\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:48.078145 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.078115    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:08:48.079952 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.079920    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-713882-65f8b499cc-8xsr2"]
Oct 31 17:08:48.674181 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.674094    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92" event={"ID":"c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa","Type":"ContainerDied","Data":"82d44711bd1909653c6161b4c30d462a7007bdfc3354f53ad10a46193e3ba17b"}
Oct 31 17:08:48.674181 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.674162    4417 scope.go:122] "RemoveContainer" containerID="cd37fd9d5f53ed581f6972686136e56d9044e468d15f3258bf2742945ac66efb"
Oct 31 17:08:48.674636 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.674115    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"
Oct 31 17:08:48.824776 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.824749    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:08:48.871069 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:48.871039    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-713882-7679498475-94m92"]
Oct 31 17:08:49.049148 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:49.049115    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" path="/var/lib/kubelet/pods/070c0b3f-387a-4f53-a0f3-87d04c736e4b/volumes"
Oct 31 17:08:49.049515 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:49.049501    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" path="/var/lib/kubelet/pods/c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa/volumes"
Oct 31 17:08:52.571937 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.571905    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:08:52.572327 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.572097    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" podUID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" containerName="maas-api" containerID="cri-o://6d971b349f3c98614e62aa80ca8ad4cae4f1b406d19c4a3fdc179201a199c18b" gracePeriod=30
Oct 31 17:08:52.697014 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.696985    4417 generic.go:356] "Generic (PLEG): container finished" podID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" containerID="6d971b349f3c98614e62aa80ca8ad4cae4f1b406d19c4a3fdc179201a199c18b" exitCode=0
Oct 31 17:08:52.697199 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.697037    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" event={"ID":"b6a3e7f0-2d2d-4840-85e4-6600011149dc","Type":"ContainerDied","Data":"6d971b349f3c98614e62aa80ca8ad4cae4f1b406d19c4a3fdc179201a199c18b"}
Oct 31 17:08:52.825039 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.824988    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:08:52.845908 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.845884    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") pod \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") "
Oct 31 17:08:52.846022 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.845972    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\" (UniqueName: \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\") pod \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") "
Oct 31 17:08:52.846081 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.846067    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\") pod \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") "
Oct 31 17:08:52.846125 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.846098    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\") pod \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\" (UID: \"b6a3e7f0-2d2d-4840-85e4-6600011149dc\") "
Oct 31 17:08:52.846457 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.846431    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca" pod "b6a3e7f0-2d2d-4840-85e4-6600011149dc" (UID: "b6a3e7f0-2d2d-4840-85e4-6600011149dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:08:52.847989 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.847961    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb" pod "b6a3e7f0-2d2d-4840-85e4-6600011149dc" (UID: "b6a3e7f0-2d2d-4840-85e4-6600011149dc"). InnerVolumeSpecName "kube-api-access-7xdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:08:52.848086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.848029    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs" pod "b6a3e7f0-2d2d-4840-85e4-6600011149dc" (UID: "b6a3e7f0-2d2d-4840-85e4-6600011149dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:08:52.848086 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.848068    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls" pod "b6a3e7f0-2d2d-4840-85e4-6600011149dc" (UID: "b6a3e7f0-2d2d-4840-85e4-6600011149dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 17:08:52.948163 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.948129    4417 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-maas-api-tls\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:52.948163 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.948165    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xdxb\" (UniqueName: \"kubernetes.io/projected/b6a3e7f0-2d2d-4840-85e4-6600011149dc-kube-api-access-7xdxb\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:52.948348 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.948180    4417 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a3e7f0-2d2d-4840-85e4-6600011149dc-openshift-service-ca\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:52.948348 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:52.948196    4417 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6a3e7f0-2d2d-4840-85e4-6600011149dc-metrics-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:08:53.702781 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:53.702749    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"
Oct 31 17:08:53.703227 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:53.702785    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l" event={"ID":"b6a3e7f0-2d2d-4840-85e4-6600011149dc","Type":"ContainerDied","Data":"833c9f372d8e7c7a3f53b5046a3a7e6adc5c5d3e79fd418851c64abf09c926b0"}
Oct 31 17:08:53.703227 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:53.702866    4417 scope.go:122] "RemoveContainer" containerID="6d971b349f3c98614e62aa80ca8ad4cae4f1b406d19c4a3fdc179201a199c18b"
Oct 31 17:08:53.854060 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:53.854025    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:08:53.900127 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:53.900093    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-713882-84bfbff65c-vxp7l"]
Oct 31 17:08:55.047686 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:08:55.047649    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" path="/var/lib/kubelet/pods/b6a3e7f0-2d2d-4840-85e4-6600011149dc/volumes"
Oct 31 17:09:01.351188 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:01.351158    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:09:01.351708 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:01.351403    4417 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy" containerID="cri-o://1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1" gracePeriod=30
Oct 31 17:09:02.943599 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:09:02.943565    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:09:06.582892 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.582867    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:09:06.683489 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683427    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683525    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683545    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683564    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683592    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683657 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683636    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683678    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683714    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:09:06.683957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683735    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683797    4417 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\") pod \"1712191b-d721-4426-855e-40a22e6c566a\" (UID: \"1712191b-d721-4426-855e-40a22e6c566a\") "
Oct 31 17:09:06.683957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683852    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.683979    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684050    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684093    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684171    4417 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1712191b-d721-4426-855e-40a22e6c566a-istiod-ca-cert\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684186    4417 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-certs\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684197    4417 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-data\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.684228 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684216    4417 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-credential-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.684523 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.684234    4417 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-workload-socket\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.685829 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.685796    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 17:09:06.685948 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.685884    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:09:06.686179 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.686163    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 17:09:06.686222 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.686191    4417 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4" pod "1712191b-d721-4426-855e-40a22e6c566a" (UID: "1712191b-d721-4426-855e-40a22e6c566a"). InnerVolumeSpecName "kube-api-access-tsxq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 17:09:06.764460 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.764429    4417 generic.go:356] "Generic (PLEG): container finished" podID="1712191b-d721-4426-855e-40a22e6c566a" containerID="1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1" exitCode=0
Oct 31 17:09:06.764603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.764471    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerDied","Data":"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"}
Oct 31 17:09:06.764603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.764547    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd" event={"ID":"1712191b-d721-4426-855e-40a22e6c566a","Type":"ContainerDied","Data":"0979394b1c755b7dbb603b391043e186e2cc3d57a5ed9e43bb096df95f44fa4d"}
Oct 31 17:09:06.764603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.764477    4417 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"
Oct 31 17:09:06.764603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.764568    4417 scope.go:122] "RemoveContainer" containerID="1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"
Oct 31 17:09:06.773378 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.773363    4417 scope.go:122] "RemoveContainer" containerID="1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.780636    4417 scope.go:122] "RemoveContainer" containerID="1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:09:06.780899    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1\": container with ID starting with 1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1 not found: ID does not exist" containerID="1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.780924    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1"} err="failed to get container status \"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1\": rpc error: code = NotFound desc = could not find container \"1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1\": container with ID starting with 1c17754d644f286b9706813e37bffd9a3bc81f96cb314579bd10b6813186aea1 not found: ID does not exist"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.780939    4417 scope.go:122] "RemoveContainer" containerID="1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:09:06.781146    4417 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7\": container with ID starting with 1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7 not found: ID does not exist" containerID="1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"
Oct 31 17:09:06.781184 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.781161    4417 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7"} err="failed to get container status \"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7\": rpc error: code = NotFound desc = could not find container \"1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7\": container with ID starting with 1d3a44551a17f37c8117b03bb7e30f295ad14c14d12d897566b0a8ae7f87d3d7 not found: ID does not exist"
Oct 31 17:09:06.785647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.785612    4417 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-istio-token\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.785703 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.785650    4417 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1712191b-d721-4426-855e-40a22e6c566a-istio-podinfo\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.785703 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.785660    4417 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxq4\" (UniqueName: \"kubernetes.io/projected/1712191b-d721-4426-855e-40a22e6c566a-kube-api-access-tsxq4\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.785703 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.785668    4417 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1712191b-d721-4426-855e-40a22e6c566a-istio-envoy\") on node \"ip-10-0-140-84.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 17:09:06.915295 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.915266    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:09:06.918873 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:06.918849    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-713882-openshift-default-6bc7bd5ff7-kcbmd"]
Oct 31 17:09:07.047719 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:07.047687    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1712191b-d721-4426-855e-40a22e6c566a" path="/var/lib/kubelet/pods/1712191b-d721-4426-855e-40a22e6c566a/volumes"
Oct 31 17:09:29.040087 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:29.040058    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"]
Oct 31 17:09:29.085921 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:29.085889    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-669wz"]
Oct 31 17:09:31.048628 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:31.048582    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb8a1a2-5c72-4b29-a126-8dfb84705795" path="/var/lib/kubelet/pods/efb8a1a2-5c72-4b29-a126-8dfb84705795/volumes"
Oct 31 17:09:32.943150 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:09:32.943116    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:09:51.051684 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:51.051653    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"]
Oct 31 17:09:51.097729 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:51.097694    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fhzlz"]
Oct 31 17:09:53.047656 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:09:53.047610    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02" path="/var/lib/kubelet/pods/a3fedf30-bdc6-4e2a-8e2c-aa02aa988a02/volumes"
Oct 31 17:10:02.943849 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:10:02.943809    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:10:23.047665 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:23.047636    4417 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"]
Oct 31 17:10:23.093145 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:23.093113    4417 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317514eb-njgfg"]
Oct 31 17:10:25.047284 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:25.047253    4417 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c202bc34-a773-4b3b-b8e8-f11b473518d2" path="/var/lib/kubelet/pods/c202bc34-a773-4b3b-b8e8-f11b473518d2/volumes"
Oct 31 17:10:25.271213 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:25.271182    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:10:25.852927 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:25.852897    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-m4nbt_c73e7d4e-3bee-4d9c-af5a-7f63e560806a/server/0.log"
Oct 31 17:10:27.113434 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:27.113402    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67bf6fc4b5-djngp_679f9b33-6b0d-419d-98e5-5ca6f4c0513f/maas-api/0.log"
Oct 31 17:10:30.736546 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:30.736516    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c4h8w_138006b7-3c26-4668-b8ab-eb13c70406a2/kuadrant-console-plugin/0.log"
Oct 31 17:10:31.060880 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:31.060854    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-m5whb_ace48c49-3306-4f3d-a16a-4f90922eea83/manager/0.log"
Oct 31 17:10:32.943116 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:10:32.943082    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:10:33.358996 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:33.358962    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-pfr86_3d30e826-29bf-4bbc-929e-3767701bb10c/kube-auth-proxy/0.log"
Oct 31 17:10:34.042885 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:34.042851    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8965v_21201683-d4a2-4c03-a84f-ff1803240663/istio-proxy/0.log"
Oct 31 17:10:34.535673 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:34.535583    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-dx8zr_ec81237f-b4d2-4591-8e0d-43c7a20df439/istio-proxy/0.log"
Oct 31 17:10:35.493734 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:35.493701    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-s8bll_312a05f7-0e52-4001-88c9-b40cc107b30a/istio-proxy/0.log"
Oct 31 17:10:35.988558 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:35.988524    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-x6tl6_ba404bf8-c02c-4e49-b4a8-41968ddb20b1/istio-proxy/0.log"
Oct 31 17:10:36.643025 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:36.642986    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:10:37.044725 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:37.044698    4417 scope.go:122] "RemoveContainer" containerID="5377466b50a3486e8587db356030ab11f658cc28f0ab99396f61a18521080b63"
Oct 31 17:10:37.053251 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:37.053228    4417 scope.go:122] "RemoveContainer" containerID="e929efdbb5d1c906c5ca11572d3cfaedb9e80f2e70067c16bb55b8fd9ef91d34"
Oct 31 17:10:37.946387 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:37.946355    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh_d59fac8e-9759-44e0-9ace-6d21585d03d3/main/0.log"
Oct 31 17:10:38.222040 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:38.221958    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f98mh_71f2171c-108c-4ae4-aad3-d0bc9cf1cd61/main/0.log"
Oct 31 17:10:38.717750 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:38.717709    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-89jt4_eb823ecd-bf57-48c0-a47a-9b0536b6c531/main/0.log"
Oct 31 17:10:38.983683 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:38.983578    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p_cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc/main/0.log"
Oct 31 17:10:51.014462 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:51.014379    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:10:51.434864 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:51.434833    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-m4nbt_c73e7d4e-3bee-4d9c-af5a-7f63e560806a/server/0.log"
Oct 31 17:10:52.493773 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:52.493721    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67bf6fc4b5-djngp_679f9b33-6b0d-419d-98e5-5ca6f4c0513f/maas-api/0.log"
Oct 31 17:10:55.812451 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:55.812419    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c4h8w_138006b7-3c26-4668-b8ab-eb13c70406a2/kuadrant-console-plugin/0.log"
Oct 31 17:10:56.030050 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:56.030021    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-m5whb_ace48c49-3306-4f3d-a16a-4f90922eea83/manager/0.log"
Oct 31 17:10:57.761276 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:57.761246    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-pfr86_3d30e826-29bf-4bbc-929e-3767701bb10c/kube-auth-proxy/0.log"
Oct 31 17:10:58.440673 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:58.440639    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8965v_21201683-d4a2-4c03-a84f-ff1803240663/istio-proxy/0.log"
Oct 31 17:10:58.995925 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:10:58.995857    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-dx8zr_ec81237f-b4d2-4591-8e0d-43c7a20df439/istio-proxy/0.log"
Oct 31 17:11:00.003262 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:00.003229    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-s8bll_312a05f7-0e52-4001-88c9-b40cc107b30a/istio-proxy/0.log"
Oct 31 17:11:00.538825 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:00.538792    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-x6tl6_ba404bf8-c02c-4e49-b4a8-41968ddb20b1/istio-proxy/0.log"
Oct 31 17:11:01.246912 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:01.246869    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:11:02.074387 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:02.074342    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s4xnh_d59fac8e-9759-44e0-9ace-6d21585d03d3/main/0.log"
Oct 31 17:11:02.333573 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:02.333461    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-f98mh_71f2171c-108c-4ae4-aad3-d0bc9cf1cd61/main/0.log"
Oct 31 17:11:02.778212 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:02.778125    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-89jt4_eb823ecd-bf57-48c0-a47a-9b0536b6c531/main/0.log"
Oct 31 17:11:02.944896 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:11:02.944864    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:11:03.050127 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:03.050102    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8dxj5p_cb6afbca-dbe9-4895-ad67-7c55c0d3f6bc/main/0.log"
Oct 31 17:11:31.252023 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:31.251987    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:11:31.819470 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:31.819436    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:11:32.564082 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:32.564056    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:11:32.791029 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:32.790999    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:11:32.943966 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:11:32.943874    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:11:37.074462 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:37.074427    4417 scope.go:122] "RemoveContainer" containerID="4538d7a731995fcf60ed67e387d78bde830a5387cada742f2f29caf705427bc6"
Oct 31 17:11:37.082576 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:11:37.082551    4417 scope.go:122] "RemoveContainer" containerID="398d19122301e29c89758295f994b549229891791265b0f71b86277853cff263"
Oct 31 17:12:02.943316 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:12:02.943279    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:12:08.476367 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:08.476334    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:12:09.061368 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:09.061339    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c95c64d65-mkxsh_d37a89b0-513a-4eff-be99-6758193b06b0/manager/0.log"
Oct 31 17:12:09.810300 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:09.810271    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:12:10.034861 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.034827    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9f4p2_49b03b4c-9d0a-41c0-8f6d-171923978a8c/payload-processing/0.log"
Oct 31 17:12:10.440249 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440218    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvx92/must-gather-mv9d2"]
Oct 31 17:12:10.440610 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440597    4417 cpu_manager.go:414] "containerMap: removing container" podUID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" containerName="maas-api"
Oct 31 17:12:10.440610 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440611    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" containerName="maas-api"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440640    4417 cpu_manager.go:414] "containerMap: removing container" podUID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" containerName="payload-processing"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440649    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" containerName="payload-processing"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440665    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440670    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440681    4417 cpu_manager.go:414] "containerMap: removing container" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440686    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440702    4417 cpu_manager.go:414] "containerMap: removing container" podUID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" containerName="payload-pre-processing"
Oct 31 17:12:10.440747 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440707    4417 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" containerName="payload-pre-processing"
Oct 31 17:12:10.441035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440764    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="c938ccbf-0ba5-4e1c-bce3-65a3ac7087fa" containerName="payload-pre-processing"
Oct 31 17:12:10.441035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440773    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a3e7f0-2d2d-4840-85e4-6600011149dc" containerName="maas-api"
Oct 31 17:12:10.441035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440779    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="070c0b3f-387a-4f53-a0f3-87d04c736e4b" containerName="payload-processing"
Oct 31 17:12:10.441035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440788    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.441035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.440904    4417 memory_manager.go:365] "RemoveStaleState removing state" podUID="1712191b-d721-4426-855e-40a22e6c566a" containerName="istio-proxy"
Oct 31 17:12:10.443799 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.443781    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.466644 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.466607    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/must-gather-mv9d2"]
Oct 31 17:12:10.492176 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.492151    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"openshift-service-ca.crt\""
Oct 31 17:12:10.492176 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.492169    4417 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"kube-root-ca.crt\""
Oct 31 17:12:10.492331 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.492219    4417 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kvx92\"/\"default-dockercfg-6qmmp\""
Oct 31 17:12:10.568455 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.568423    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-must-gather-output\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.568667 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.568483    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92znp\" (UniqueName: \"kubernetes.io/projected/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-kube-api-access-92znp\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.669486 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.669448    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92znp\" (UniqueName: \"kubernetes.io/projected/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-kube-api-access-92znp\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.669723 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.669558    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-must-gather-output\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.669957 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.669937    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-must-gather-output\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.727650 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.727560    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92znp\" (UniqueName: \"kubernetes.io/projected/b9fb7e57-3b99-4fcb-8da4-c80d0809405c-kube-api-access-92znp\") pod \"must-gather-mv9d2\" (UID: \"b9fb7e57-3b99-4fcb-8da4-c80d0809405c\") " pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:10.756603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:10.756570    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/must-gather-mv9d2"
Oct 31 17:12:11.121067 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:11.121032    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/must-gather-mv9d2"]
Oct 31 17:12:11.128464 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:12:11.128434    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9fb7e57_3b99_4fcb_8da4_c80d0809405c.slice/crio-95f199d12cf16c11bce5c61222ef3f43e57ff27eb13a0636cc9332cd130ae7f0 WatchSource:0}: Error finding container 95f199d12cf16c11bce5c61222ef3f43e57ff27eb13a0636cc9332cd130ae7f0: Status 404 returned error can't find the container with id 95f199d12cf16c11bce5c61222ef3f43e57ff27eb13a0636cc9332cd130ae7f0
Oct 31 17:12:11.130369 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:11.130347    4417 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 17:12:11.721536 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:11.721501    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/must-gather-mv9d2" event={"ID":"b9fb7e57-3b99-4fcb-8da4-c80d0809405c","Type":"ContainerStarted","Data":"95f199d12cf16c11bce5c61222ef3f43e57ff27eb13a0636cc9332cd130ae7f0"}
Oct 31 17:12:12.730712 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:12.730589    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/must-gather-mv9d2" event={"ID":"b9fb7e57-3b99-4fcb-8da4-c80d0809405c","Type":"ContainerStarted","Data":"a2608119fb87af1000bb9d3f5096db6b7c6ffa738cecbb745612375d97070b62"}
Oct 31 17:12:13.737164 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:13.737130    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/must-gather-mv9d2" event={"ID":"b9fb7e57-3b99-4fcb-8da4-c80d0809405c","Type":"ContainerStarted","Data":"016ad7d1733a68e0a72f9242dcb48c1597c1f29b7e1c23586ad795dd272629c3"}
Oct 31 17:12:13.842669 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:13.842591    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kvx92/must-gather-mv9d2" podStartSLOduration=2.450021974 podStartE2EDuration="3.842576871s" totalImagesPullingTime="1.392554897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:12:13.84075077 +0000 UTC m=+4178.902325957" watchObservedRunningTime="2026-10-31 17:12:13.842576871 +0000 UTC m=+4178.904152057"
Oct 31 17:12:18.866317 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:18.866221    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4p5d6_a029ca40-3196-4f27-b72a-1d2a2b76197e/global-pull-secret-syncer/0.log"
Oct 31 17:12:22.673687 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:22.673649    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n76gt_29a5a550-ba91-4cd4-92a7-bb61db0e3eb5/konnectivity-agent/0.log"
Oct 31 17:12:23.556728 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:23.556694    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-84.us-west-2.compute.internal_102c36941ddb017d6ed417f277ac7f2a/haproxy/0.log"
Oct 31 17:12:29.241838 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.241799    4417 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"]
Oct 31 17:12:29.249581 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.249551    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.303441 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.303406    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"]
Oct 31 17:12:29.370369 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.370325    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-lib-modules\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.370603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.370387    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjtw\" (UniqueName: \"kubernetes.io/projected/bb475b79-a2e4-4553-a124-2bc50ca7a166-kube-api-access-wgjtw\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.370603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.370538    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-proc\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.370603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.370580    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-sys\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.370603 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.370598    4417 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-podres\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471384 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471348    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-proc\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471384 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471391    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-sys\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471412    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-podres\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471466    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-lib-modules\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471488    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-proc\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471498    4417 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjtw\" (UniqueName: \"kubernetes.io/projected/bb475b79-a2e4-4553-a124-2bc50ca7a166-kube-api-access-wgjtw\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471516    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-sys\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471647 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471574    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-podres\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.471976 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.471741    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb475b79-a2e4-4553-a124-2bc50ca7a166-lib-modules\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.524196 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.524127    4417 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjtw\" (UniqueName: \"kubernetes.io/projected/bb475b79-a2e4-4553-a124-2bc50ca7a166-kube-api-access-wgjtw\") pod \"perf-node-gather-daemonset-q77gz\" (UID: \"bb475b79-a2e4-4553-a124-2bc50ca7a166\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.568338 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.568308    4417 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:29.937600 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:29.937569    4417 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"]
Oct 31 17:12:29.944904 ip-10-0-140-84 kubenswrapper[4417]: W1031 17:12:29.944876    4417 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb475b79_a2e4_4553_a124_2bc50ca7a166.slice/crio-359dd543c9529ed530dd94680c269965e6fc32b22a855c9fd0ce73898047012f WatchSource:0}: Error finding container 359dd543c9529ed530dd94680c269965e6fc32b22a855c9fd0ce73898047012f: Status 404 returned error can't find the container with id 359dd543c9529ed530dd94680c269965e6fc32b22a855c9fd0ce73898047012f
Oct 31 17:12:30.615035 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:30.615007    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c4h8w_138006b7-3c26-4668-b8ab-eb13c70406a2/kuadrant-console-plugin/0.log"
Oct 31 17:12:30.858816 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:30.858779    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz" event={"ID":"bb475b79-a2e4-4553-a124-2bc50ca7a166","Type":"ContainerStarted","Data":"ea0117894bbb8421c419ed86f4f90a7e6c5b660d093a1e9cd883fef2687f8d4a"}
Oct 31 17:12:30.859139 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:30.858838    4417 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz" event={"ID":"bb475b79-a2e4-4553-a124-2bc50ca7a166","Type":"ContainerStarted","Data":"359dd543c9529ed530dd94680c269965e6fc32b22a855c9fd0ce73898047012f"}
Oct 31 17:12:30.859139 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:30.858877    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:30.964668 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:30.964533    4417 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz" podStartSLOduration=1.964515262 podStartE2EDuration="1.964515262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:12:30.962819185 +0000 UTC m=+4196.024394371" watchObservedRunningTime="2026-10-31 17:12:30.964515262 +0000 UTC m=+4196.026090451"
Oct 31 17:12:31.065598 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:31.065561    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-m5whb_ace48c49-3306-4f3d-a16a-4f90922eea83/manager/0.log"
Oct 31 17:12:32.943482 ip-10-0-140-84 kubenswrapper[4417]: E1031 17:12:32.943447    4417 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g4jw4" containerName="extractor"
Oct 31 17:12:34.500576 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:34.500549    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/alertmanager/0.log"
Oct 31 17:12:34.918837 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:34.918806    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/config-reloader/0.log"
Oct 31 17:12:35.338909 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:35.338868    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/kube-rbac-proxy-web/0.log"
Oct 31 17:12:35.757224 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:35.757151    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/kube-rbac-proxy/0.log"
Oct 31 17:12:36.175307 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:36.175273    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/kube-rbac-proxy-metric/0.log"
Oct 31 17:12:36.592677 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:36.592648    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/prom-label-proxy/0.log"
Oct 31 17:12:36.873417 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:36.873342    4417 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-q77gz"
Oct 31 17:12:37.009087 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:37.009062    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ee602f0-74c4-43c8-a904-49ee517ce276/init-config-reloader/0.log"
Oct 31 17:12:37.996794 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:37.996744    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5q7s_e721f7ad-6663-4c39-bce7-9cbc649ec162/kube-state-metrics/0.log"
Oct 31 17:12:38.409237 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:38.409207    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5q7s_e721f7ad-6663-4c39-bce7-9cbc649ec162/kube-rbac-proxy-main/0.log"
Oct 31 17:12:38.826974 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:38.826950    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r5q7s_e721f7ad-6663-4c39-bce7-9cbc649ec162/kube-rbac-proxy-self/0.log"
Oct 31 17:12:40.093059 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:40.093032    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8npgm_a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6/node-exporter/0.log"
Oct 31 17:12:40.509954 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:40.509888    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8npgm_a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6/kube-rbac-proxy/0.log"
Oct 31 17:12:40.928287 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:40.928261    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8npgm_a8c36f19-c8ef-4f4f-b58f-b871bf33f0c6/init-textfile/0.log"
Oct 31 17:12:48.905017 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:48.904962    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nfltz_a42dd32c-24e9-4e10-841e-ff0055744450/prometheus-operator-admission-webhook/0.log"
Oct 31 17:12:52.795101 ip-10-0-140-84 kubenswrapper[4417]: I1031 17:12:52.795062    4417 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-744f97f697-pfr86_3d30e826-29bf-4bbc-929e-3767701bb10c/kube-auth-proxy/0.log"